Debugging the AI Engine Application
You can debug the AI Engine graph application in a standalone capacity, to run and debug just the AI Engine. Or you can debug the system project, including the top-level PS application as well as the AI Engine graph. Within this framework, you can also debug applications built from the command line, or system projects built in the Vitis™ IDE. You can debug applications running on the Linux OS, or in bare-metal systems. Finally, you can debug hardware emulation builds that let you simulate the application, or debug the actual application running on hardware. All of these configurations are addressed in the following topics.
Launching Debug from the Vitis IDE
As described in Using the Vitis IDE you can build a system level project incorporating the kernels running in the AI Engine domain, the PL domain, and an application running in the PS domain. These system-level projects can be debugged together, form within the Vitis IDE, or you can debug the AI Engine graph application, focusing just on that particular problem. This section discusses running the Debug Environment from the Vitis IDE for hardware emulation and hardware builds.
Using printf for Event Tracing
The simplest form of tracing is to use a formatted printf()
statement in the code for printing debug
messages. Visual inspection of intermediate values, addresses, etc. can help you
understand the progress of program execution. No additional include files are
necessary for using printf()
other than standard
C/C++ includes (stdio.h). You can add printf()
statements to your code to be processed during
simulation, or hardware emulation, and remove them or comment them out for hardware
builds.
Adding printf
statements to your
AI Engine kernel code will increase the
compiled size of the AI Engine program. Be
careful that the compiled size of your kernel code does not exceed the per-AI Engine processor memory limit of 16 KB.
aiesimulator --profile
command to enable
the printf()
execution during a simulator run. If
--profile
is not specified, the printf()
function is ignored.A separate driver and binary is used for this functionality to allow
the main simulator to remain as fast as possible. Using the debug simulator driver
produces a per-tile profile report under the output directory which gives detailed
cycle-level statistics of kernel execution. In addition, using the --profile
option generates a run_summary file that is written to the ./aiesimulator_output folder that can be viewed as
described in Viewing the Run Summary in the Vitis Analyzer.
Hardware Emulation Debug from the Vitis IDE
- Start the QEMU emulation environment by selecting the
This launches the emulator and then waits until Linux is booted within the QEMU. The Emulation console shows a transcript of the QEMU launch and Linux boot process. You can tell when the process has completed when the progress dialog closes and the
qemu%
prompt is black. You can examine the transcript for details of this process.When launching hardware emulation, you can specify options for the AI Engine simulator that runs the graph application, as described in Simulator Options for Hardware Emulation, The options can be specified in the Emulator Arguments field shown in the prior figure by specifying the following command:
-aie-sim-options ../aiesim_options.txt
command. - Right-click on the top-level system project, and select the Debug Configurations dialog box. command. This opens the
- Press Debug to proceed.
This opens the Debug perspective in the Vitis IDE, and connects to the PS application and AI Engine graph running on their respective cores in the QEMU. The application automatically breaks at the
main()
function for all the ELF files.
Hardware Debug from the Vitis IDE
- Burn <project>/Hardware/package/sd_card.img to a physical SD card. This creates a boot-able medium for your target platform.
- Insert the SD card into the card reader of the VCK190 evaluation kit.
- Change the boot-mode settings of the card to SD boot mode, and power up the board.
- After the VCK190 is booted, enter the
mount
command at the command prompt to get a list of mount points. As shown in the following figure, themount
command displays mounting information for the system.TIP: Be sure to capture the proper path for thecd
command in the next step, and subsequent commands, based on the results of themount
command. - Execute the following commands:
cd /mnt/mmcblk0p1 source init.sh cat /etc/xocl.txt
The
cat
command will display the platform namexilinx_vck190_base_202110_1
. You can confirm it is the same as your specified platform and that your setup is correct. - Run
ifconfig
to get the IP address of the target card. The IP address is used to set up a TCF agent connection in Vitis IDE. The target needs to connect to the network assigned IP address. - Create a target connection to the remote accelerator card. Select the Target Connections view. command to open the
- In the Target Connections view, right-click on the Linux TCF Agent and select the New Target command to open the New Target Connection dialog box.
- Specify the Target
Name, enable the Set as
default target check box, and specify the Host IP address of the accelerator
card that you obtained in an earlier step.
- Click OK to close the dialog box and continue.
- Right-click on the top-level system project and select the
This opens the Debug Configurations dialog box to let you set up the tool. For the Hardware builds, you will need to create two launch configurations: one for the top-level system project, and a second for the PS application.
command. - In the Debug
Configurations dialog box select the New Launch Configuration ()
command to open the Debug
Configurations dialog box as shown.
Be sure to set the following fields on the dialog box as shown in the preceding figure.
- Remote Working Directory: Specify the remote mount location from the accelerator card as determined in an earlier step.
- Linux TCF Agent: Select the new agent you built with the specified IP address for the accelerator card.
- Disable build before launch: This is necessary because without this step, the tool will try to build your system before running the application.
- Select Apply to apply your
changes and select Debug to start the
process.
This opens the Debug perspective in the Vitis IDE, and connects to the PS application and AI Engine graph running on their respective cores in the QEMU. The application automatically breaks at the
main()
function for all the ELF files.
Bare-metal Debug from the Vitis IDE
With the bare-metal system project built as described in Building a Bare-metal AI Engine in the Vitis IDE, you must use the following steps to debug both the AI Engine graph, together with the bare-metal PS application on the Hardware build target.
This process is a bit more complex than debugging a Linux application in the Vitis IDE. In that case the PS host application was built as a part of the top-level system project. Here the PS application is built as a standalone project, and must be separately included in the debug configurations when launching the debug environment. This is detailed in the following steps.
- Right-click on the top-level system project and select the
This opens the Debug Configurations dialog box to let you set up the tool.
IMPORTANT: For the Hardware build, you will need to create two Debug configurations: one for the top-level system project, and a second for the bare-metal PS application.
command. - In the Debug
Configurations dialog box select the New Launch Configuration ()
command to open the Debug
Configurations dialog box as shown.
Notice the following fields on the Debug Configurations dialog box:
- Project: Reflects the name of the top-level system project which includes the AI Engine graph application, the PL kernels, and the HW-Link projects.
- Hardware Server: Specifies a local connection to the board. You can configure this differently for a remotely connected board.
- Linux TCF Agent: Is disabled for bare-metal systems.
- Disable build before launch: Enable this to prevent the tool from building your system before launching the application.
- Select Apply to save and apply your changes, and select Close to close the dialog box.
- Right-click the baremetal_app
project in the Explorer view, and select as shown in the following image.
- The Single Application Debug
Configurations dialog box opens as shown in the
following figure.
- Specify a Name to identify the configuration as applying to the PS application. The Connection is set up automatically for the local hardware.
- Change to select the Target Setup tab of the Debug Configurations dialog box, and deselect the Reset entire system and Program Device check boxes in the dialog box.
- Click Apply to proceed.
- You also need to create a launch group containing the two debug
configurations you just created. The launch group lets you launch multiple
configurations as a group. Use the main toolbar menu command by selecting
Debug
Configurations dialog box, and then select Launch Group as shown. IMPORTANT: You must use the main toolbar menu Debug command because it provides the complete set of Debug Configuration options rather than the reduced options found in the Explorer or Assistant views, which are limited by your selections.
Click on Launch Group, and provide a Name for the new group. Click Add the top-level project and PS application Debug configurations to the group as shown in the previous image. Click OK to create the launch group.
to open the - The Launch Group is displayed with both debug configurations added. Click Debug to launch the Debug perspective.
- Because you are launching both the top-level project and PS debug configurations on a single target connection, the Vitis IDE will display a Launch Config Conflict message. Click No on the conflict message to proceed.
- The Debug
perspective is opened, as shown in the following image. Because there are two
debug configurations, you will see two instances of the PS application. You can
ignore the second instance of the PS application.
Launching Debug from the Command Line
Debugging projects built from the command line is a special challenge because
the various elements of the system, the compiled graph application (libadf.a), the device binary (XCLBIN), and the
top-level application (host.cpp), must be
gathered together and presented as a system. In the Vitis IDE this is done through the system-level project that is
created when you first created the graph project. In the command line flow, this
service is provided by the Vitis compiler
package process. As described in Packaging,
the v++ --package
command gathers the elements of
the system together to create a boot container for either hardware emulation or
hardware builds.
In addition, you must separately manage the different elements of the debug
environment, running the QEMU emulation environment, or running the xrt_server
for connecting to the hardware, and
launching the Vitis IDE to run the debug
process. The steps and requirements for command line debug are detailed in the
following topics.
Hardware Emulation Debug from the Command Line
- Launch the QEMU emulator environment using the launch_hw_emu.sh script, that is generated
during the
--package
process. - Launch the Vitis IDE in
standalone debug mode using the
vitis -debug
option. - Configure the debug environment to connect to the PS and AI Engine domains within the system.
- For AI Engine platforms, the files
required for emulation of the system are defined by the
--package
command, including the emulation script. To launch the emulation environment use the following command from your build directory:./emulation/launch_hw_emu.sh -pid-file emulation.pid -no-reboot \ -add-env ENABLE_RDWR_DEBUG=true -add-env RDWR_DEBUG_PORT=10100 -forward-port 1440 1534
Where:
- ./emulation is the output directory of the package process as described in Packaging, and is where the launch_hw_emu.sh script can be found.
-add-env RDWR_DEBUG_PORT=${aie_mem_sock_port}
defines the port for communicating with the AI Engine domain. In the previous example, it is 10100.-forward-port ${linux_tcf_agent_port} 1534
defines the port for the Linux TCF agent. In the previous example, it is 1440, which is the default.
TIP: Any free ports can be used foraie_mem_sock_port
andlinux_tcf_agent_port
in the above command template. However, these port are mandatory for enabling the AI Engine application and Linux application debug respectively.This command launches the emulator and then waits until Linux is booted within the QEMU. The QEMU shell shows a transcript of the QEMU launch and Linux boot process. You can tell when the process has completed when the
qemu%
prompt is displayed. At that time you are ready to proceed. - Run the following command in the QEMU shell at the
qemu%
prompt:source /mnt/sd-mmcblk0p1/init.sh
Note: The mount drive is reported in your QEMU transcript, or can be found by typingmount
at theqemu%
prompt. The drive above should be standard for Xilinx platforms. - In a second terminal window, launch the XRT server application
using the following
command:
xrt_server -I300 -S -s tcp::4352
where:
-I300
defines an idle timeout, in which the server quits if there is no response.-S
specifies print server properties in JSON format to stdout.-s tcp::${xrt_server_port}
defines the agent listening protocol and port. In the previous example, it is 4352, but can be any free port.
- Create a Tcl script with the name aie_app_debug.tcl to set up the AI Engine debug
environment:
#Set up the required environment # The aie_mem_socket and xrt_server ports must match what was specified in earlier commands. set aie_work_dir "<AIE_Project>/Work" set aie_mem_sock_port "10100" set xrt_server_port "4352" set app_name "aie_graph" #Echo the environment setup puts "Vitis install: $XILINX_VITIS" puts "Application: $app_name, Work Directory: $aie_work_dir" puts "XRT Server Port: $xrt_server_port, AIE Port: $aie_mem_sock_port" #Set up AIE Debug environment set source_tcl_cmd "source $XILINX_VITIS/scripts/vitis/util/aie_debug_init.tcl" puts "$source_tcl_cmd" eval $source_tcl_cmd ##run the command to connect and display debug targets set aie_debug_cmd "init_aie_debug -work-dir $aie_work_dir -url tcp::$xrt_server_port \ -memsock-url localhost:$aie_mem_sock_port -sim-type memserver -name $app_name -full-program" puts "$aie_debug_cmd" eval $aie_debug_cmd
Note: This script requires the $XILINX_VITIS environment variable to be set up. - After the QEMU environment and xrt_server are up and running, you
can launch the Vitis IDE in stand-alone
debug mode in a third terminal
window:
vitis -debug -flow embedded_accel -target hw_emu -exe ./ps_app \ -program-args ${xcl_bin_dir}/binary_container_1.xclbin -port 1440
where:
vitis -debug
- Launches the Vitis IDE in stand-alone debug mode.
-flow embedded_accel
- Specifies the embedded processor application acceleration flow.
-target hw_emu
- Indicates the target build being debugged.
-exe ./ps_app
- Indicates the PS application to run and debug.
-program-args ${xcl_bin_dir}/binary_container_1.xclbin
- Refers to the location of the XCLBIN file to be loaded as an argument to the executable.
-port 1440
- Specifies the
${linux_tcf_agent_port}
as discussed previously.
This opens the Vitis IDE with the Debug perspective displayed, and the debug configuration for the PS application loaded.
- In the Debug perspective of the Vitis IDE, create a new target connection of type
Hardware Server
with the name
aie_xrt_server
. Specifylocalhost
as host, andxrt_server_port
, 4352 in the previous example, as the port. - Create a new Debug configuration of type Single Application Debug as
shown.
- Debug Type
Attach to running target
- Connection
aie_xrt_server
- Execute Script
- Specify the path to aie_app_debug.tcl defined in Step 5.
- Press Debug to proceed.
This connects to the PS application and AI Engine graph running on their respective cores in the QEMU. The application automatically breaks at the
main()
function for all the ELF files.
Debugging Only the AI Engine Graph
To debug just the AI Engine domain without the PS, follow steps 1 to 4 in Hardware Emulation Debug from the Command Line. Then, instead of step 5, continue by using the following step:
- After the QEMU environment and xrt_server are up and running, you can launch the
Vitis IDE in standalone debug mode
in a third terminal
window:
vitis -debug -flow embedded -os baremetal -port 4352 \ -launch-script <aie_project>/aie_app_debug.tcl
where:
vitis -debug
- Launches the Vitis IDE in standalone debug mode.
-flow embedded
- Specifies the embedded processor flow for the AI Engine processor.
-os baremetal
- For bare-metal OS of the AI Engine domain.
-port 4352
- Specifies the xrt_server port as discussed in Step 3.
-launch_script <aie_project>/aie_app_debug.tcl
- Specifies the Tcl script from Step 4, which sets up the AI Engine debug environment.
This opens the Vitis IDE with the Debug perspective displayed, and the debug configuration for the AI Engine application loaded.
Hardware Debug from the Command Line
- Launch the Vitis IDE in
standalone debug mode using the
vitis -debug
option. - Connect to the board using the hardware server (
hw_server
) command. - Configure the debug environment for the AI Engine domain.
- Burn <project>/Hardware/package/sd_card.img to a physical SD card. This creates a bootable medium for your target platform.
- Insert the SD card into the card reader of the VCK190 evaluation kit.
- Change the boot-mode settings of the card to SD boot mode, and power up the board.
- Run
ifconfig
to get the IP address of the target card. The IP address is used to set up a the hardware serve connection in Vitis IDE. The target needs to connect to the network assigned IP address. - In a terminal window, launch the hardware server to connect to the board.
- Create a Tcl script with the name aie_app_debug.tcl to set up the AI Engine debug
environment:
#Set up the required environment # The aie_mem_socket and xrt_server ports must match what was specified in earlier commands. set aie_work_dir "<AIE_Project>/Work" set hw_server_host "gandalf" set app_name "aie_graph" #Printing the information puts "Install: $$XILINX_VITIS" puts "Application: $app_name, Work Directory: $aie_work_dir" puts "Hardware Server: $hw_server_host" set source_tcl_cmd "source ${vitis_install}/scripts/vitis/util/aie_debug_init.tcl" puts "$source_tcl_cmd" eval $source_tcl_cmd ##run the command to connect and display debug targets set aie_debug_cmd "init_aie_debug -work-dir $aie_work_dir -url tcp:$hw_server_host:3121 -jtag -name $app_name" puts "$aie_debug_cmd" eval $aie_debug_cmd
Note: This script requires setting up the $XILINX_VITIS environment variable. - After the board system and hw_server are up and running, launch the Vitis IDE in standalone debug mode in a second terminal
window:
vitis -debug -flow embedded_accel -target hw -exe ./ps_app \ -program-args ${xcl_bin_dir}/binary_container_1.xclbin -host ${linux_tcf_agent_host} \ -port 1534
where:
vitis -debug
- Launches the Vitis IDE in standalone debug mode.
-flow embedded_accel
- Specifies the embedded processor application acceleration flow.
-target hw
- Defines the hardware build as being debugged.
-exe ./ps_app
- Indicates the PS application to run and debug.
-program-args ${xcl_bin_dir}/binary_container_1.xclbin
- Refers to the location of the XCLBIN file to be loaded as an argument to the executable.
-host ${linux_tcf_agent_host}
- Specifies the host name or IP address obtained
from Linux running on the board,
${linux_tcf_agent_host}
. -port 1534
- Specifies the port the Linux TCF agent is running
on,
${linux_tcf_agent_port}
. In this case 1534.
This opens the Vitis IDE with the Debug perspective displayed, and the debug configuration for the PS application loaded.
- Create a new debug configuration of type Single Application Debug as
shown.
- Click Debug to proceed.
This connects to the PS application and AI Engine graph running on their respective cores in the QEMU. The application automatically breaks at the
main()
function for all the ELF files.
Debugging Only the AI Engine Hardware
To debug just the AI Engine domain without the PS, follow steps 1 to 6 in Hardware Debug from the Command Line. Then, instead of step 7, continue by using the following step:
- After the board and the hw_server are up and running, launch the Vitis IDE in stand-alone debug mode in a separate terminal
window:
vitis -debug -flow embedded -os baremetal -host gandalf \ -launch-script <aie_project>/aie_app_debug.tcl
Where:
vitis -debug
- Launches the Vitis IDE in standalone debug mode.
-flow embedded
- Specifies the embedded processor flow for the AI Engine processor.
-os baremetal
- For the baremetal OS of the AI Engine domain.
-host gandalf
- Specifies the host name of the hw_server (see Step 5 of Hardware Debug from the Command Line).
-launch_script <aie_project>/aie_app_debug.tcl
- Specifies the Tcl script from Step 6 of Hardware Debug from the Command Line, which sets up the AI Engine debug environment.
This opens the Vitis IDE, displays the Debug perspective, and loads the debug configuration for the AI Engine application.
Using the Debug Environment
The Vitis IDE debug environment has many features found in traditional GUI-based debug environments, such as GDB. You can add break points to the code, step over or step into specific lines of codes, loops, or functions, and examine the state of variables and force them to specific values. These are just a few of the features in the Vitis IDE debug environment.
After you have launched the Debug perspective, you will see several windows or views displayed, such as the Debug view in the upper left of the display, as shown in the following figure. During the debug process, several windows show the debug state that include code at break point, step over states, break-points view, variables view, registers view, disassemble view, and pipeline view (single kernel only).
The Debug view shows states of cores that are under debug. It shows where (which file and which line of source code of the file) debugger stops at and with what action it is taking (breakpoint/step over/…) as shown in the following figure.
The following figure shows the Breakpoints information with current setup breakpoints. The square with a check mark indicates the breakpoint is enabled. Click on the check mark to clear the check mark and disable breakpoint during debugging. This lets you manage breakpoints without having to remove them or add them back into the code.
main()
by default.
Breakpoints attached to a while
statement consume
two breakpoint resources. A workaround is to attach the breakpoint inside the
while
loop. This only consumes one
breakpoint.The Variables view shows the kernel variables values. Depending on variable type, clicking on a variable shows its type, value, and the address of the variable. For array/structure variables, click on the arrow of the variable expands array/structure content of the array.
Click on a variable to obtain address information and then click on the + from the Memory view and enter the memory address of the variable. The value of that variable shows up along with other memory content. Variable values can be changed by click on variable’s value field, enter new value and enter again to set the value of the variable.
You can specify the format of the data presented in the Memory window by clicking on the New Renderings tab, specifying the format of the data to present, and click Add Rendering(s).
In the Registers view, the values are updated during the debug process and are highlighted in the Vitis IDE.
The debug control commands let you step into/over/return from the source code. Other commands let you resume, stop, terminate, or disconnect the debugger from the Vitis IDE.
The Vitis IDE supports multi-kernel debug and multi-domain (PS and AI Engine) debug. Depending on the application, there can be hundreds of kernels being debugged. Having granular control of each core as well as all cores within one domain is important. Select one core from the Vitis IDE and click the Resume icon to resume debug execution for the core only. Select one level above all cores within the AI Engine domain, and click the Resume icon to resume all AI Engine core executions. Resuming execution for all the kernels in the graph is dependent on a variety of conditions such as the availability of data to each of the kernels to avoid stalls, or the setup of individual kernel breakpoints. You should also ensure that the kernels not being debugged are free to run.
main()
function is created
automatically by the AI Engine compiler for each
core that wraps around the kernel implementation. The Debug view displays the complete source
code including the inserted portion.Viewing Data from Window Interfaces
As described in Window and Streaming Data API the data flows into and out of an AI Engine kernel through access windows, or a stream interface. During debug you might want to see the value of these data access windows as data is passing through the kernels. In the following figure you can trace the series of objects that you must work through to examine the contents of the variables.
- In the code example shown at Step 1 in the figure, you can see that the variable
cb_input
is a pointer of typeinput_window_cint16
. This shows the declaration of an input window carrying complex integers where the real part is 16-bits wide, and the imaginary part is 16 bits. - Move to the
cb_input
variable in the Variables view. This is the pointer representation of the data access window that holds the input data for the kernel. However, the kernel functions merely operate on pointers to the window data structures passed to them as arguments. The input window data structure is defined by a pointer calledptr
. Thisptr
is defined by the input data window API as described in Window Operations for Kernels. - Examine the contents of the variable
ptr
. It is an address with value 0x00038000. - In the Memory window, click on the
+
sign to input the address 0x38000 as shown in Step 4. - The Memory window displays the content at address of 0x38000.
- This is the data contained in the data access window defined by the
cb_input
variable. The previous example has a 64-byte margin size, so the actual data starts from 0x38000 plus 0x40 = 0x38040. You can examine the data contents, display it in a specific data format, or copy it to the clipboard and export it to a separate file as described in Using the Debug Environment.Notice the HEX values, such as F4D9ED71, presented in the Memory window as shown in the previous figure represent a
cint16
type complex number with a 16-bit signed integer real number and a 16-bit signed imaginary number with the following values read from an input file:-4751 -2855 -1107 -5121 1062 -2732 4465 4323 9899 5093 5289 -2426 -1063 -5425 ...
Viewing Data from Stream Interfaces
Data flows into and out of an AI Engine kernel through access windows, or a stream interface. During debug you might want
to see the value of these data access windows as data is passing through the
kernels. In the case of data windows, the Vitis
IDE debug environment provides methods to view and access the data as described in
Viewing Data from Window Interfaces. In the case of stream
interface connections, it is recommended to add printf()
statements to your code to let you examine the data passing
through the kernel.
printf()
statements to the code suppresses compiler
optimizations, and results in a larger kernel executable program that might not fit
into the available memory of the AI Engine processor.When adding the printf()
statement in your
kernel code you must also add the --profile
option
in the Run Configurations
or Debug Configurations
dialog box in the Vitis IDE. Add --profile
to the Arguments tab of the Debug Configuration, along with
whatever other options are already specified, as shown in the following figure.
Adding the printf()
statements to your source
code, results in the output of streaming data as it is processed by the kernel. The
following figure shows an example of such output in the console window. This
provides visibility to capture and debug the dataflow through streaming
interfaces.
Pipeline View for Single Kernel Debug
The AI Engine Pipeline view in the Vitis IDE allows you to correlate instructions executed in a specific clock cycle with the labels in the Disassembly view. The underlying AI Engine pipeline is exposed in debug mode using the pipeline view. The Vitis IDE only supports pipeline view for graphs containing single kernels.
To enable the Pipeline view on graphs with a single kernel, select Generate Profile from the project debug configuration after the project has been built successfully.
- If your graph has multiple tiles this view does not appear; however the case with multiple kernels in a single tile is supported.
- The pipeline view support is available with the AI Engine simulator only.
Click on Debug to start debugging the application. Note the Pipeline view shows up automatically in the Debugger Console window.
Run-time statistics of the kernel in the pipeline view are highlighted in the previous figure.
- AI Engine kernel cycle count
- Program counter
- ID = Instruction decode
- E1-E7 are the AI Engine execution stages. Almost all operations in the scalar unit are scheduled in E1 stage of the pipeline besides non-linear operations. The vector unit scheduling spans from the ID stage to the E6 stage. Address Generation Units (AGUs) span over two pipeline stages. The address is ready in the E2 stage of the pipeline. For load units, the data will be available in the AI Engine from the memory module in the E7 stage. For the store unit, the data will be sent out from the AI Engine to the memory module in the E5 or E6 stage of the pipeline depending on the type of instruction.