3.1 Host to External M
Note: For more information about the EMIF SS, please refer to Memory Subsystem Intel FPGA IP User Guide
Large buffers of data are usually transferred between host and on-board memory in oneAPI applications. This necessitates a Direct Memory Access(DMA) Engine between host and on-board memory. In oneAPI ASP designs for OFS reference platform, this DMA engine is placed in the AFU region.
-As described in section 2.1.5.1, there are different configurations for memories on board. In addition to above, figure 1-3 also shows an additional IP in the host to memory datapath, called Memory Bank Divider. This IP is used for handling one of the most commonly used configurations, i.e. configuring multiple memory banks of same type as a contiguous memory region. In this case, the kernel has a contiguous view of the memory and data can be interleaved across the different memory channels. The host must also have the same view of the memory in order to ensure read and write transactions from correct addresses.
+As described in section 2.1.5.1, there are different configurations for memories on board. In addition to above, figure 1-3 also shows an additional IP in the host to memory datapath, called Memory Bank Divider. This IP is used for handling one of the most commonly used configurations, i.e. configuring multiple memory banks of same type as a contiguous memory region. In this case, the kernel has a contiguous view of the memory and data can be interleaved across the different memory channels. The host must also have the same view of the memory in order to ensure read and write transactions from correct addresses.
3.1.1 Memory Bank Divider
The OpenCL™ Memory Bank Divider is a Platform Designer component that takes an incoming request from the host interface on the Avalon®-MM agent port and routes it to the appropriate bank host port. This component must reside on the path between the host and the global memory interfaces. In addition, it must reside outside of the path between the kernel and the global memory interfaces.
Figure 3-1: Memory Bank Divider IP
+
+
+
+
Table 3-1: Parameter Settings for the Memory Bank Divider Component
@@ -3468,15 +3514,24 @@ The host exchanges control signals with kernel with the help of an additional IP . The control signals coming from the host are on a different clock domain (PCIe clock) while the kernel runs on different clock frequency . The Kernel Interface IP handles the clock domain crossing for these control signals as well as handles communication with kernel CSR, interrupts, generates the reset for kernel.
+
The host exchanges control signals with kernel with the help of an additional IP . The control signals coming from the host are on a different clock domain (PCIe clock) while the kernel runs on different clock frequency . The Kernel Interface IP handles the clock domain crossing for these control signals as well as handles communication with kernel CSR, interrupts, generates the reset for kernel.
All oneAPI ASP designs must instantiate Kernel Interface IPs to ensure the kernel functions correctly.
3.2.1 Kernel Interface
The Kernel Interface is a Platform Designer component that allows the host interface to access and control the oneAPI kernel.
Figure 3-2: Kernel Interface IP
+
+
+
+
+
+
+
+
Table 3-3: Parameter Settings for the Kernel Interface Component
@@ -3575,7 +3630,7 @@ 4.1 Memory Mapped Device(MMD)
Free memory if allocation done in step 4
Close device. No further operations permitted until subsequent open device call
-Table below summarizes all APIs listed in aocl_mmd.h.
+Table below summarizes all APIs listed in aocl_mmd.h.
Table 4-1: Summary of MMD API from aocl_mmd.h
@@ -3671,6 +3726,8 @@ 4.1 Memory Mapped Device(MMD)
+
+
Sections below cover more details for each API (expected arguments, return values). Section 5.4 discusses more about the implementation of the MMD layer APIs in oneAPI ASPs for OFS reference platforms.
4.1.1 aocl_mmd_get_offline_info
@@ -3733,6 +3790,9 @@ 4.1.1 aocl_mmd_get_offline_
+
+
-
param_value_size: Size of the param_value field in bytes. This size_t value should match the size of the expected return type that the enum definition indicates.
@@ -3886,7 +3946,7 @@
4.1.2 aocl_mmd_get_infoparam_size_ret: A pointer argument of type size_t* that receives the number of bytes of returned data.
-Capability Values
+Capability Values
Table 4-4: Capability Values for aocl_mmd_get_info Function
@@ -4037,7 +4097,7 @@ 4.1.6 aocl_mmd_
Where:
exception_type acts as a bitfield that contains exactly one bit, corresponding to an exception number.user_private_info and user_cb represent pointers to binary data that the OpenCL implementation return. These pointers log additional information that is helpful for debugging the error.user_private_info and user_cb represent pointers to binary data that the OpenCL implementation return. These pointers log additional information that is helpful for debugging the error.
user_data: The void* type user-provided data that passes to fn when it is called.4.1.17 aocl_mmd_host_allocalignment: The alignment (in bytes) of the allocation.
-
-
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
+properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
-
error: The error code defined by AOCL_MMD_ERROR*:
@@ -4519,7 +4579,7 @@ 4.1.19 aocl_mmd_device_allocalignment: The alignment (in bytes) of the memory region.
-
-
properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
+properties: Specifies additional information about the allocated memory, described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
Return Value
@@ -4559,7 +4619,7 @@ 4.1.20 aocl_mmd_shared_allocalignment: The alignment (in bytes) of the allocation.
-properties: Specifies additional information about the allocated memory described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
+properties: Specifies additional information about the allocated memory described by a property type name and its corresponding value. Each property type name is immediately followed by the corresponding desired value. The list is terminated with a zero. For example, [, , , , 0]
error: The error code defined by AOCL_MMD_ERROR*.
@@ -4574,6 +4634,8 @@ 4.1.20 aocl_mmd_shared_alloc
Return Value
If the aocl_mmd_shared_alloc function executes successfully, the return value is a valid pointer value. Otherwise, the return value is NULL.
+
+
4.1.21 aocl_mmd_shared_migrate
@@ -4679,6 +4741,8 @@ 5.1 oneapi-asp D
As described in section 2.0, oneAPI compiler & runtime use the board_env.xml and board_spec.xml files to get information about the FPGA platform. The compiler expects the board_env.xml file to be at the topmost level in the platform directory. The board_env.xml file describes the location of the hardware files & platform libraries.
Figure 5-1: Sample board_env.xml File
+
+
Figure 5-1 shows a sample board_env.xml file, the corresponding oneAPI ASP directory structure must match the following format. Table 5-1 provides details on each folder.
In addition to below folders, oneAPI ASP for OFS reference platforms have another folder for the software layer (MMD & board utilities) source code located in common directory (oneapi-asp/common). This is because a common source code is utilized for both d5005 and n6001 reference platform ASPs.
@@ -4861,6 +4925,8 @@ 5.2 oneapi-asp Build Flow
The following figure shows the oneapi-asp build process.
Figure 5-2: oneapi-asp Build Flow
+
+
Table 5-5 Environment Variables used in Build Flow
@@ -4974,12 +5040,22 @@ 5.3 oneapi-aspNote: Please see oneapi-asp/Platform-Name/hardware folder for the board variants for each OFS reference platform.
Figure 5-3: oneapi-asp Reference Platform Hardware Design - Board Variant #1
+
+
+
+
Figure 5-4: oneapi-asp Reference Platform with USM Hardware Design - Board Variant #2
+
+
Figure 5-5: oneapi-asp Reference Platform with IO Pipes Hardware Design - Board Variant #3
+
+
Figure 5-6: oneapi-asp Reference Platform with IO Pipes and USM Hardware Design - Board Variant #4
+
+
The ofs_plat_afu.sv is the top level entity for the oneapi-asp.
All hardware design files for the components inside ofs_plat_afu module are inside the oneapi-asp/Platform-Name/hardware/Board-Variant/build directory. The FIM database files imported during oneapi-asp build (section 5.2) are located in oneapi-asp/Platform-Name/hardware/Board-Variant/fim_platform directory.
@@ -5039,6 +5115,12 @@ 5.3 oneapi-asp
+
+
+
+
+
+
The hardware implementation diagrams show a PF/VF Mux/De-mux module in the AFU region. The PF/VF mux routes packets to AFU component based on pf_num and vf_num information in PCIe TLP header. For more information about the PF/VF mux, please refer to the PF/VF Mapping details in FPGA Interface Manager Technical Reference Manual for your target device (links are available in section 1.3).
The oneapi-asp resides inside the AFU region and ,depending on the FIM configuration, connects to the lowest PF or lowest VF number that routes into the PR slot. Table below summazires the PF/VF mapping in oneapi-asp for some of the different FIM configurations.
Table 5-8: PF/VF Mapping in oneapi-asp
@@ -5076,7 +5158,7 @@ 5.3 oneapi-asp
*Note: For more information on different FIM configurations & how to compile these, please refer to FPGA Interface Manager Developer Guide for Open FPGA Stack for your target device (links are available in section 1.3).
-Sections below provide some more information on some blocks in the hardware design block diagram shown above. Refer to section 3.1 for more information about Memory Bank Divider and to section 3.2 for information about Kernel Interface.
+Sections below provide some more information on some blocks in the hardware design block diagram shown above. Refer to section 3.1 for more information about Memory Bank Divider and to section 3.2 for information about Kernel Interface.
@@ -5089,6 +5171,8 @@ 5.3.2 Direct Memory Access(D
The Direct Memory Access module is located in the host to EMIF datapath in the oneapi-asp and provides a controller to execute transfers from host to DDR on the board and vice versa. The source files for the DMA module used in oneapi-asp for OFS reference platforms are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/rtl/dma directory. Figure below shows the DMA module interface.
Figure 5-7: DMA Controller Block Diagram
+
+
Figure 5-3 shows the instantiation of this dma_top module as dma_controller_inst. Table 5-9 shows the signal names and descriptions. Section 5-5 covers the complete compilation flow.
Table 5-9: DMA Module Signal Descriptions
@@ -5144,13 +5228,15 @@ 5.3.2 Direct Memory Access(D
5.3.3 User Datagram Protocol(UDP) Engine
-I/O pipes allow kernel to stream data directly using HSSI. To demonstrate this functionality, reference design in oneapi-asp repository (refer to Figure 5-5 and 5-6) has a UDP protocol engine to allow transmitting UDP/IP packets over HSSI..
+I/O pipes allow kernel to stream data directly using HSSI. To demonstrate this functionality, reference design in oneapi-asp repository (refer to Figure 5-5 and 5-6) has a UDP protocol engine to allow transmitting UDP/IP packets over HSSI..
Figure below shows a simple block diagram of the UDP engine.
Figure 5-8: UDP Offload Engine
+
+
The UDP engine consists of a separate receive (rx) and trasmit (tx) path. The following functionalilty is performed by this reference design engine:
-- Implements an Address Resolution Protocol (ARP) functionality to respond to be able to send & respond to ARP requests. This is needed for routing between different subnets using a gateway.
+- Implements an Address Resolution Protocol (ARP) functionality to respond to be able to send & respond to ARP requests. This is needed for routing between different subnets using a gateway.
- Packetizes data from kernel to add the required header information (for MAC, IP & UDP layers)
- Extracts data from packets received by removing header information
- Handles clock crossing between kernel clock and Ethernet MAC clock domains
@@ -5159,14 +5245,20 @@ 5.3.3 User Datagram Protoc
Note: The same engine is used in the board variant with USM shown in Figure 5-6 (source files are in oneapi-asp/n6001/hardware/ofs_n6001_usm_iopipes/build/rtl/udp_offload_engine).
+
+
5.3.4 Hardware Compile Flow
Figure below shows the compile flow overview; the oneAPI compiler generated hardware circuit is compiled by Intel® Quartus® software along with design files for oneapi-asp.
Figure 5-9: oneAPI Compile Flow Overview
+
+
The oneAPI compiler uses the board_spec.xml to get more information about the oneapi-asp configuration. board_spec.xml file has a compile element to allow control of the Intel® Quartus® software compilation flow. The attributes of this element are discussed in section 2.1.2. The oneapi-asp uses tcl scripts to control the Intel® Quartus® software compilation flow. Figure 5-10 shows the flow and scripts used. All compilation scripts are located in oneapi-asp/Platform-Name/hardware/Board-Variant/build/scripts folder.
Figure 5-10: Compilation Scripts in oneapi-asp
+
+
Table 5-10 summarizes notes for reference numbers 1-5 marked in figure above.
Table 5-10: Notes for Reference Numbers in Figure 5-10
@@ -5188,6 +5280,8 @@ 5.3.4 Hardware Compile Flow<
+
+
5.4 oneapi-asp Memory Mapped Device(MMD) Layer Implementation
@@ -5307,6 +5401,8 @@ 5.4
+
+
The build flow scripts build the MMD library, i.e. libintel_opae_mmd, and place them in oneapi-asp/Platform-Name/linux64/lib folder. The MMD library is specified as part of mmdlib, linklibs element in board_env.xml and used at runtime (refer to figure 5-1 for sample board_env.xml file and section 2.2 for more information about board_env.xml elements).
Use of OPAE library in MMD
@@ -5416,12 +5512,14 @@ 
