Building the Device Binary

Generating kernels from C/C++ code for use in the Vitis core development kit follows the standard Vitis HLS process. However, because the kernel is required to operate in the Vitis software platform, the standard kernel requirements must be satisfied (see Kernel Properties). Most importantly, the interfaces must be modeled as AXI memory interfaces, except for scalar parameters which are mapped to an AXI4-Lite interface. Vitis HLS automatically defines the interface ports to meet the standard kernel requirements when using the Vitis Bottom Up Flow as described here.

The process for creating and compiling your HLS kernel is outlined briefly below. You should refer to Creating a New Vitis HLS Project in the Vitis HLS Flow documentation for a more complete description of this process.

1. Launch Vitis HLS to open the integrated design environment (IDE), and specify File > New Project.
2. In the New Vitis HLS Project wizard, specify the Project name, define the Location for the project, and click Next.
3. In the Add/Remove Files page, click Add Files to add the kernel source code to the project. Select Top Function to define the kernel function by clicking the Browse button, and click Next when done.
4. You can specify a C-based simulation test bench if you have one available, by clicking Add Files, or skip this by clicking Next.
   TIP: As discussed in the Vitis HLS documentation, the use of a test bench is strongly recommended.
5. In the Solution Configuration page, you must specify the Clock Period for the kernel.
   
   
6. Choose the target platform by clicking the browse button (…) in the Part Selection field to open the Device Selection Dialog box. Select Boards, and select the target platform for your compiled kernel, as shown below. Click OK to select the platform and return to the Solution Configuration page.
7. In the Solution Configuration page, select the Vitis Kernel Flow Target drop-down menu under Flow Target, and click Finish to complete the process and create your HLS kernel project.
   IMPORTANT: You must select the Vitis Kernel Flow Target to generate the Xilinx object (XO) file from the project.

When the HLS project has been created you can Run C-Synthesis to compile the kernel code. Refer to the Vitis HLS documentation for a complete description of the HLS tool flow.

After synthesis is completed, the kernel can be exported as an XO file for use in the Vitis core development kit. The export command is available through the Solution > Export RTL command from the main menu.

Specify the file location, and the kernel is exported as a Xilinx object XO file.

The (XO) file can be used as an input file during the v++ linking process. Refer to Linking the Kernels for more information. You can also add it to an application project in the Vitis IDE, as discussed in Creating a Vitis IDE Project.

However, keep in mind that HLS kernels, created in the bottom-up flow described here, have certain limitations when used in the Vitis application acceleration development flow. Software emulation is not supported for applications using HLS kernels, because duplicated header file dependencies can create issues. GDB debug is not supported in the hardware emulation flow for HLS kernels, or RTL kernels.

By default, the linker builds a single hardware instance from a kernel. If the host program will execute the same kernel multiple times, due to data processing requirements for instance, then it must execute the kernel on the hardware accelerator in a sequential manner. This can impact overall application performance. However, you can customize the kernel linking stage to instantiate multiple hardware compute units (CUs) from a single kernel. This can improve performance as the host program can now make multiple overlapping kernel calls, executing kernels concurrently by running separate compute units.

Multiple CUs of a kernel can be created by using the connectivity.nk option in the v++ config file during linking. Edit a config file to include the needed options, and specify it in the v++ command line with the --config option, as described in Vitis Compiler Command.

[connectivity]
#nk=<kernel name>:<number>:<cu_name>.<cu_name>...
nk=vadd:2

Where:

<kernel_name>

Specifies the name of the kernel to instantiate multiple times.

<number>

The number of kernel instances, or CUs, to implement in hardware.

<cu_name>.<cu_name>...

Specifies the instance names for the specified number of instances. This is optional, and the CU name will default to kernel_1 when it is not specified.

v++ --config vadd_config.cfg ...

In the vadd example above, the result is two instances of the vadd kernel, named vadd_1 and vadd_2.

[connectivity]
nk=vadd:3:vadd_X.vadd_Y.vadd_Z

The link phase is when the memory ports of the kernels are connected to memory resources which include DDR, HBM, and PLRAM. By default, when the xclbin file is produced during the v++ linking process, all kernel memory interfaces are connected to the same global memory bank (or gmem). As a result, only one kernel interface can transfer data to/from the memory bank at one time, limiting the performance of the application due to memory access.

While the Vitis compiler can automatically connect CU to global memory resources, you can also manually specify which global memory bank each kernel argument (or interface) is connected to. Proper configuration of kernel to memory connectivity is important to maximize bandwidth, optimize data transfers, and improve overall performance. Even if there is only one compute unit in the device, mapping its input and output arguments to different global memory banks can improve performance by enabling simultaneous accesses to input and output data.

The following example is based on the Kernel Interfaces example code. Start by assigning the kernel arguments to separate bundles to increase the available interface ports, then assign the arguments to separate memory banks:

1. In C/C++ kernels, assign arguments to separate bundles in the kernel code prior to compiling them:

   void cnn( int *pixel, // Input pixel
     int *weights, // Input Weight Matrix
     int *out, // Output pixel
     ... // Other input or Output ports
   		   
   #pragma HLS INTERFACE m_axi port=pixel offset=slave bundle=gmem
   #pragma HLS INTERFACE m_axi port=weights offset=slave bundle=gmem1
   #pragma HLS INTERFACE m_axi port=out offset=slave bundle=gmem
   

   Note that the memory interface inputs pixel and weights are assigned different bundle names in the example above, while out is bundled with pixel. This creates two separate interface ports.

   IMPORTANT: You must specify bundle= names using all lowercase characters to be able to assign it to a specific memory bank using the --connectivity.sp option.
2. Edit a config file to include the --connectivity.sp option, and specify it in the v++ command line with the --config option, as described in Vitis Compiler Command.
   For example, for the cnn kernel shown above, the connectivity.sp option in the config file would be as follows:

   [connectivity]
   #sp=<compute_unit_name>.<argument>:<bank name> 
   sp=cnn_1.pixel:DDR[0]          
   sp=cnn_1.weights:DDR[1]
   sp=cnn_1.out:DDR[2]
   

   Where:

   • <compute_unit_name> is an instance name of the CU as determined by the connectivity.nk option, described in Creating Multiple Instances of a Kernel, or is simply <kernel_name>_1 if multiple CUs are not specified.
   • <argument> is the name of the kernel argument. Alternatively, you can specify the name of the kernel interface as defined by the HLS INTERFACE pragma for C/C++ kernels, including m_axi_ and the bundle name. In the cnn kernel above, the ports would be m_axi_gmem and m_axi_gmem1.
     TIP: For RTL kernels, the interface is specified by the interface name defined in the kernel.xml file.
   • <bank_name> is denoted as DDR[0], DDR[1], DDR[2], and DDR[3] for a platform with four DDR banks. You can also specify the memory as a contiguous range of banks, such as DDR[0:2], in which case XRT will assign the memory bank at run time.

     Some platforms also provide support for PLRAM, HBM, HP or MIG memory, in which case you would use PLRAM[0], HBM[0], HP[0] or MIG[0]. You can use the platforminfo utility to get information on the global memory banks available in a specified platform. Refer to platforminfo Utility for more information.

     In platforms that include both DDR and HBM memory banks, kernels must use separate AXI interfaces to access the different memories. DDR and PLRAM access can be shared from a single port.

   • IMPORTANT: Customized bank assignments might also need to be reflected in the host code in some cases, as described in Assigning DDR Bank in Host Code.

[connectivity]
## Syntax
##sp=<cu_name>.<interface_name>:HOST[0]
sp=cnn_1.m_axi_gmem:HOST[0]

The Vitis core development kit supports streaming data transfer between two kernels, allowing data to move directly from one kernel to another without having to transmit back through global memory. However, the process has to be implemented in the kernel code itself, as described in Streaming Data in User-Managed Never-Ending Kernels, and also specified during the kernel build process.

The streaming data ports of kernels can be connected during v++ linking using the --connectivity.sc option. This option can be specified at the command line, or from a config file that is specified using the --config option, as described in Vitis Compiler Command.

To connect the streaming output port of a producer kernel to the streaming input port of a consumer kernel, setup the connection in the v++ config file using the connectivity.stream_connect option as follows:

[connectivity]
#stream_connect=<cu_name>.<output_port>:<cu_name>.<input_port>:[<fifo_depth>]
stream_connect=vadd_1.stream_out:vadd_2.stream_in

Where:

• <cu_name> is an instance name of the CU as determined by the connectivity.nk option, described in Creating Multiple Instances of a Kernel.
• <output_port> or <input_port> is the streaming port defined in the producer or consumer kernel.
• [:<fifo_depth>] inserts a FIFO of the specified depth between the two streaming ports to prevent stalls. The value is specified as an integer.

Currently, Xilinx devices on Data Center accelerator cards use stacked silicon consisting of several Super Logic Regions (SLRs) to provide device resources, including global memory. For best performance, when assigning ports to global memory banks, as described in Mapping Kernel Ports to Memory, it is best that the CU instance is assigned to the same SLR as the global memory it is connected to. In this case, you will want to manually assign the kernel instance, or CU into the same SLR as the global memory to ensure the best performance.

A CU can be assigned to an SLR using the connectivity.slr option in a config file. The syntax of the connectivity.slr option in the config file is as follows:

[connectivity]
#slr=<compute_unit_name>:<slr_ID>
slr=vadd_1:SLR2
slr=vadd_2:SLR3

Where:

• <compute_unit_name> is an instance name of the CU as determined by the connectivity.nk option, described in Creating Multiple Instances of a Kernel, or is simply <kernel_name>_1 if multiple CUs are not specified.
• <slr_ID> is the SLR number to which the CU is assigned, in the form SLR0, SLR1,...

The assignment of a CU to an SLR must be specified for each CU separately, but is not required. If an assigned CU is connected to global memory located in another SLR, the tool will automatically insert SLR crossing registers to help with timing closure. In the absence of an SLR assignment, the v++ linker is free to assign the CU to any SLR.

v++ -l --config config_slr.cfg ...

Generally, in embedded processor platforms, and also in newer Data Center accelerator cards, the device binary can connect multiple kernels to the platform with different clock frequencies. Each kernel, or unique instance of the kernel can connect to a specified clock frequency, or multiple clocks, and different kernels can use different clock frequencies generated by the platform.

During the compilation process (v++ -c), you can specify a kernel frequency using the --hls.clock command. This lets you compile the kernel targeting the specified frequency, and lets the Vitis HLS tool perform validation of the kernel logic at the specified frequency. However, this is just an implementation target for compilation, but does provide optimization and feedback.

During the linking process, when the kernels are connected to the platform to build the device binary, you can specify the clock frequency for the kernels using the --clock Options of the v++ command. Therefore the process for managing clock frequencies is as follows:

1. Compile the HLS code at a specified frequency using the Vitis compiler:

   v++ -c -k <krnl_name> --hls.clock freqHz:<krnl_name>

   TIP: freqHz must be in Hz (for example, 250000000Hz is 250 MHz).
2. During linking, specify the clock frequency or clock ID for each clock signal in a kernel with the following command:

   v++ -l ... --clock.freqHz <freqHz>:kernelName.clk_name

You can specify the --clock option using either a clock ID from the platform shell, or by specifying a frequency for the kernel clock. When specifying the clock ID, the kernel frequency is defined by the frequency of that clock ID on the platform. When specifying the kernel frequency, the platform attempts to create the specified frequency by scaling one of the available fixed platform clocks. In some cases, the clock frequency can only be achieved in some approximation, and you can specify the --clock.tolerance or --clock.default_tolerance to indicate an acceptable range. If the available fixed clock cannot be scaled within the acceptable tolerance, a warning is issued and the kernel is connected to the default clock.

For challenging designs, it can take multiple iterations of Vivado implementation using multiple different strategies to achieve timing closure. This topic shows you how to launch multiple implementation strategies at the same time in the hardware build (-t hw), and how to identify and use successful runs to generate the device binary and complete the build.

As explained in --vivado Options the --vivado.impl.strategies command enables you to specify multiple strategies to run in a single build pass. The command line would look as follows:

v++ --link -s -g -t hw --platform xilinx_zcu102_base_202010_1 -I . \
--vivado.impl.strategies "Performance_Explore,Area_Explore" -o kernel.xclbin hello.xo

In the example above, the Performance_Explore and Area_Explore strategies are run simultaneously in the Vivado build to see which returns the best results. You can specify the ALL to have all available strategies run within the tool.

You can also determine this option in a configuration file in the following form:

#Vivado Implementation Strategies
[vivado]
impl.strategies=Performance_Explore,Area_Explore

The Vitis compiler automatically picks the first completed run results that meets timing to proceed with the build process and generate the device binary. However, you can also direct the tool to wait for all runs to complete and pick the best results from the completed runs before proceeding. This would require the use of the --advanced.compiler directive as shown:

[advanced]
param=compiler.multiStrategiesWaitOnAllRuns=1

compiler.multiStrategiesWaitOnAllRuns=0 represents the default behavior. If you want v++ to wait for all runs to complete, which will get their report files, change that parameter value to 1.

As discussed in Link Summary: Multiple Strategies and Timing Reports, Vitis analyzer displays the implementation results for the all strategies that have been allowed to run to completion. This includes an overview of the implementation results, as well as a Timing Summary report. You can use this feature to review the different strategies and results.

You can also manually review the results of all implementation strategies after they have completed. Then, use the results of any of the implementation runs by using the --reuse_impl option as described in Using -to_step and Launching Vivado Interactively.