/*
====== Getting Started with Vitis AI (Under Construction) ======

<WRAP todo important>
== Under Construction ==
Sources used in this guide are not currently public.
</WRAP>

<WRAP info>
== For now it only covers legacy DNNDK examples as the full Vitis AI library requires deploying the DPU as a Vitis acceleration kernel and is only compatible with Zynq UltraScale ==
</WRAP>

===== Introduction =====
Vitis AI is Xilinx's development stack for implementing accelerated AI inference on their hardware platforms such as the Zynq 7000 and Zynq UltraScale. It consists of the DPU IP core, a software library for developing applications, software tools for training, quantization and deployment of AI models and a large set of prebuilt AI models and example projects.

This guide walks through the process of installing the development stack, building and deploying an accelerated application on a Digilent board.

----
===== Prerequisites =====
  * Ubuntu 18.04 LTS or another Linux distribution supported by Petalinux 2020.1
  * Petalinux 2020.1
  * Docker (Installation steps are located here: https://docs.docker.com/engine/install/)
  * 50+ GB of free hard drive space.

----
===== Guide =====
==== Downloading the support project and building the Petalinux SDK ===
//In order to build the DNNDK library and the applications, the Petalinux SDK is required to be built and installed.//

<WRAP group> <WRAP column half>
Clone the umbrella git repository for your chosen board and checkout the branch for the AI support project:
<code>
git clone https://github.com/Digilent/Zybo-Z7.git
cd Zybo-Z7
git checkout 20/AI/master
</code>
**NOTE**: The given commands are for the Zybo Z7-20 AI project without Pcam 5C support.
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Source the Petalinux Tools and build the SDK
<code>
source <Petalinux_Install>/settings.sh
cd os
petalinux-build --sdk
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Navigate to the images/linux folder and install the SDK
<code>
cd images/linux
./sdk.sh -d <install_path>
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
==== Downloading Vitis-AI and building the DNNDK library ===
<WRAP group> <WRAP column half>
Clone the Vitis-AI repository and checkout version 1.2.1
<code>
git clone https://github.com/Xilinx/Vitis-AI.git
cd Vitis-AI
git checkout v1.2.1
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
//In order to configure the library to use the Linux driver instead of the XRT runtime, changes are required to be made to the Makefile.cfg and build the library//
<WRAP group> <WRAP column half>
Navigate to the DNNDK source folder and open Makefile.cfg in a text editor.
<code>
cd mpsoc/vitis_ai_dnndk_src/
gedit Makefile.cfg
</code>
Comment out line 11 by placing # at the begining of the line and uncomment line 12 by removing it, the file should look like the image and save the file. 
</WRAP> <WRAP column half>
{{:reference:programmable-logic:guides:vitis-ai_makefilecfg.png?400|}}
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Now enable the Petalinux SDK built earlier and build the library
<code>
source <sdk path>/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
OR
source <sdk path>/environment-setup-aarch64-xilinx-linux
make
make python # build the python support files
make package
</code>
You should have a new file called vitis-ai_v1.2_dnndk.tar.gz
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
==== Preparing the AI model for building the application ===
<WRAP group> <WRAP column half>
Download Xilinx's AI Model Zoo
<code>
cd <Vitis-AI repo location>/AI-Model-Zoo/
sh get_model.sh
</code>
You should have a new folder called models
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Build the Docker container for Vitis AI
<code>
cd <Vitis-AI repo location>/docker
./docker_build_cpu.sh
-OR- 
./docker_build_gpu.sh #if you have NVIDIA CUDA support
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
//Before entering the Docker container there are 2 files needed to be placed inside the Vitis-AI repository as there will be no easy access to files outside the container//

One of them is a .hwh, usually called system.hwh, which contains information about the DPU and can be found in :

  * In the AI board project repository
  * In the Vivado project : <path_to_your_project>/<name_of_project>.srcs/sources_1/bd/<name_of_bd>/hw_handoff/<name_of_bd>.hwh
If the DPU IP settings have been modified then it is required to get the file from the Vivado project as it will invalidate the one in the repository.

The other one is called arch.json and will be created with the following content
<code>
{
    "target"   : "DPUCZDX8G",
    "dcf"      : "./dpu-08-07-2020-10-30.dcf",
    "cpu_arch" : "arm32"
}
</code>
Replace arm32 with arm64 if you're using Zynq Ultrascale
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Activate the docker container using the command:
<code>
<Vitis-AI repository location>/docker_run.sh xilinx/vitis-ai-cpu:latest 
OR
<Vitis-AI repository location>/docker_run.sh xilinx/vitis-ai-gpu:latest
</code>
Accept the terms and conditions and you should be presented with a bash prompt from inside the container
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Navigate to the model folder you need, in this case I need the model resnet50 for the demo with the same name, and copy the 2 files mentioned earlier:
<code>
cd AI-Model-Zoo/models/all_models_1.2/caffe/cf_resnet50_imagenet_224_224_7.7G_1.2
cp /workspace/<your hwh file> ./
cp /workspace/arch.json ./
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Now we need to generate the DPU metadata and compile the model for the application.
If the model is made in Tensorflow(the model folder starts with tf_):
<code>
conda activate vitis-ai-tensorflow
dlet -f <your hwh file>
#Open the arch.json file and make sure that the dcf parameter matches the newly created .dcf file
vai_c_tensorflow -f quantized/deploy_model.pb -a ./arch.json -n <name_of_the_model> -o output
</code>
If the model is made in Caffe(the model folder starts with cf_):
<code>
conda activate vitis-ai-caffe
dlet -f <your hwh file>
#Open the arch.json file and make sure that the dcf parameter matches the newly created .dcf file
vai_c_caffe -p quantized/Edge/deploy.prototxt -c quantized/Edge/deploy.caffemodel -a ./arch.json -n <name_of_the_model> -o output
</code>
If it successfully compiled there should be a new folder called output and the .elf file is the compiled model. Now exit from Vitis-AI environment with 
<code>
exit
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
==== Building an DNNDK example ===
<WRAP group> <WRAP column half>
Now that the AI model is prepared to be added into an application, we need to install the DNNDK library inside the Petalinux SDK sysroot.
Go to the DNNDK library sources, extract the created archive and run the installation script:
<code>
cd <Vitis-AI repository path>/mpsoc/vitis_ai_dnndk_src/
tar -xvzf vitis-ai_v1.2_dnndk.tar.gz
cd vitis-ai_v1.2_dnndk
./install.sh <Petalinux SDK path>/sysroots/cortexa9t2hf-neon-xilinx-linux-gnueabi
-OR-
./install.sh <Petalinux SDK path>/sysroots/aarch64-xilinx-linux
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
To build the application there are two ways, using the command line or the Vitis development environment

<WRAP group> <WRAP column half>
Reactivate the Petalinux SDK in case you've closed the terminal window and navigate to the DNNDK sample you want to compile
<code>
source <Path to SDK>/environment-setup-cortexa9t2hf-neon-xilinx-linux-gnueabi
OR
source <Path to SDK>/environment-setup-aarch64-xilinx-linux
cd <Vitis-AI repository path>/mpsoc/vitis_ai_dnndk_samples/<your chosen example>
</code>
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
--> C/C++ application #

<WRAP group> <WRAP column half>
To compile the model we have to copy the compiled model and tweak the Makefile if you use a Zynq 7000 based board
<code>
mkdir model
cp <Path to your model>/output/<name_of_model_elf> ./model/
gedit Makefile
</code>
If you need to modify the Makefile, change this line
<code>
CFLAGS +=  -mcpu=cortex-a53
</code>
To:
<code>
CFLAGS +=  -mcpu=cortex-a9
</code>
And save the changes
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<WRAP group> <WRAP column half>
Now build the application:
<code>
make
</code>
You should have a new file called as your application
</WRAP> <WRAP column half>
</WRAP> </WRAP>
----
<--

--> Python application #
To run Python based application you only need to copy the model and compile it as a shared library:
<code>
cp <Path to your model>/output/<name_of_model_elf> ./
gcc -fPIC -shared your_model.elf -o libdpumodel<your_model>.so
</code>
<--

----
==== Deploying the DNNDK example ===
=== Uploading the files to the board ===
1. Plug in the Ethernet cable from a router into the board, and start the board. 

2. Start the serial console using a terminal program like TeraTerm or Putty

3. Get the ip address using "ip addr" or ifconfig

4. Upload the vitis-ai_v1.2_dnndk.tar.gz to the board: scp <path to Vitis-AI>/mpsoc/vitis_ai_dnndk_src/vitis-ai_v1.2_dnndk.tar.gz root@<board_ip>:~/

5. Upload your application executable or the python script and the .so file using scp <path to the file> root@<board_ip>:~/

6. Download the sample data provided by Xilinx [[https://www.xilinx.com/bin/public/openDownload?filename=vitis-ai_v1.2_dnndk_sample_img.tar.gz|vitis-ai_v1.2_dnndk_sample_img.tar.gz]] and upload the archive to the board

7. Connect to the board using ssh with X11 forwarding: ssh -X root@<board_ip>

----
=== Installing DNNDK and running the application ===
//The following steps are done on the board in an ssh session with X11 forwarding.//
<WRAP group> <WRAP column half>
1. Extract the DNNDK library and the sample data using tar -xvzf <name_of_archive>

2. Go to the DNNDK library folder and run the install script:
<code>
cd vitis-ai_v1.2_dnndk
./install.sh
</code>

3. Create a new folder in the sample data and copy your application there
<code>
mkdir ~/vitis-ai_v1.2_dnndk/test
cp <application files> ~/vitis-ai_v1.2_dnndk/test
</code>

4. Load the dpu driver using <code>
modprobe dpu
</code>
 Then you run the application. The resnet50 demo should look like the image
</WRAP> <WRAP column half>
{{:reference:programmable-logic:guides:vitis-ai_resnet50.png?400|}}
</WRAP> </WRAP>
----
===== References =====
  * https://github.com/Xilinx/Vitis-AI/blob/master/mpsoc/vitis_ai_dnndk_src/README.md (DNNDK Compilation)
  * https://github.com/Xilinx/Vitis-AI/blob/master/mpsoc/README.md (Compilation flow for DNNDK Examples for zcu102,104)
  * https://www.xilinx.com/html_docs/vitis_ai/1_2/runningexamples.html#vdj1571812778281 (Information about all the DNNDK examples)*
  * https://www.xilinx.com/html_docs/vitis_ai/1_2/fsb1576131639266.html (VAI_C compiler usage)*
  * https://www.xilinx.com/html_docs/vitis_ai/1_2/knk1571811540439.html (Usage for DNNDK utilities such as DLet)*
  * https://github.com/Xilinx/Vitis-AI/tree/master/AI-Model-Zoo/README.md (Vitis AI Model Zoo folder structure)
  * https://www.xilinx.com/html_docs/vitis_ai/1_2/zkj1576857115470.html
*/