Warning Note that, before we begin, we assume that you already have a vSphere data center! If you do not yet have your own data center, you can follow the vSphere documentation to lease a facility and build your own data center.

Note that this article does not discuss how to build a vSphere data center!

At the time this article was written, my friends EdgeNeko and Codgician provided me with tremendous support! Without their help, it would have been impossible for me to accomplish this on my own. I also gained a wealth of knowledge from them. Thank you so much!

While we were working on this together, they also wrote some notes. With their permission, I have quoted some of their notes, including: Configuring an Nvidia Grid Guest from Scratch on Linux. If you are interested, I strongly encourage you to follow these two experts, as they are top-tier professionals in the field of computing.

file

References

Why do this?

As is well known, AI is the future. However, AI demands an enormous amount of computational power. Traditional CPU computing can barely meet the requirements of AI workloads.

A currently popular solution is to use GPUs for AI computing. Compared to CPUs, GPUs offer higher parallel computing capabilities and more computing units, significantly accelerating large-scale AI tasks. Additionally, GPUs support various libraries and frameworks specifically developed for AI computing, such as CUDA, CuDNN, and TensorFlow. These libraries and frameworks make AI computing more efficient and easier to implement.

Besides GPUs, there are other solutions available today, such as using TPUs (Google's custom AI computing accelerators) for AI computing. However, at present, GPUs remain one of the most widely used solutions in AI computing and also have one of the most mature support ecosystems.

CUDA?

file

Compared to traditional CPU architectures, the CUDA architecture is more focused on general-purpose parallel computing, and is therefore sometimes referred to as General-Purpose Computing on Graphics Processing Units (GPGPU). CUDA transforms GPUs into highly efficient parallel computing devices, thereby accelerating many scientific, engineering, and big data analytics applications. CUDA is a high-performance parallel computing framework developed by NVIDIA, based on the C/C++ programming model. It is optimized for the specific characteristics of GPU hardware and provides numerous programming language extensions, runtime libraries, and development tools that are tightly integrated with GPU hardware.

Executing parallel computations on a GPU can significantly improve computational efficiency. By leveraging the GPU's massive parallelism and high-bandwidth memory, CUDA greatly enhances the performance of applications such as scientific computing and deep learning, resulting in superior computational efficiency and execution speed.

Although CUDA is a general-purpose parallel computing framework, not all applications can benefit from performance improvements when using CUDA.

Why can't consumer graphics cards, such as NVIDIA® GeForce RTX™ 4090, be used?

To perform GPU-based computation, you first need to purchase a professional compute card rather than a consumer-grade GeForce graphics card. This is because there are significant design differences between professional compute cards and consumer-level graphics cards. Professional compute cards, such as the Nvidia Tesla series, are specifically designed for high-performance computing. They feature more compute units and greater video memory, making them more efficient in data processing and computation compared to consumer-grade graphics cards. Additionally, professional compute cards typically support double-precision calculation, a more precise mode of computation, whereas consumer graphics cards only support single-precision calculation.

file

Professional compute cards offer several other advantages in supporting high-performance computing. One of these is the provision of GPU virtualization technology (vGPU). vGPU allows multiple virtual machines to share the same physical compute card, with each virtual machine receiving independent GPU compute performance. This effectively improves the utilization and efficiency of computing resources. This technology is particularly useful for cloud computing, virtual desktops, and remote graphics applications. Among these, sdev (Shared Device) technology is a commonly used GPU virtualization technique that enables virtual machines to share the same physical GPU and migrate across different physical hosts. The sdev technology leverages PCIe device direct assignment to partition the physical GPU into multiple virtual devices and allocate them to different virtual machines. During virtual machine migration, all virtual devices can be migrated to the new physical host, enabling seamless virtual machine migration. This enhances the scalability and portability of GPU virtualization technology.

file

Furthermore, professional compute cards can provide GRID functionality. GRID is a solution introduced by Nvidia for cloud virtualization, which delivers more efficient graphics processing and compute performance through GPU hardware acceleration. Using GRID technology can significantly improve the graphical performance and computational speed for users on virtual desktops and cloud applications, thereby enhancing user productivity. Consequently, enterprises, research institutions, and cloud computing service providers can all benefit from GRID technology.

Additionally, professional computing cards offer superior stability and reliability. They can maintain high-intensity operation for days or even months, feature longer lifespans, and provide better resistance to interference and vibration. These cards are well-suited for prolonged, high-load computing tasks, whereas consumer graphics cards are better suited for general gaming and graphics processing work.

FPGA ?

In addition to GPUs and TPUs, FPGAs (Field-Programmable Gate Arrays) are also commonly used as AI computing accelerators. An FPGA is a hardware circuit chip whose functionality can be altered through programming, making it suitable for custom hardware accelerator designs. Compared to GPUs and TPUs, FPGAs offer a higher degree of flexibility and customizability, allowing optimization based on the specific requirements of applications and algorithms. At the same time, FPGAs generally have relatively lower power consumption and latency. However, the development cost and difficulty for FPGAs are high, as they require proficiency in hardware description languages and circuit design knowledge. Therefore, using GPUs and TPUs for AI computing remains the more prevalent choice currently.

We will not discuss FPGAs in depth here.

SR-IOV?

file

SR-IOV (Single Root I/O Virtualization) is another hardware virtualization technology. Similar to the sdev technology in GPU virtualization, it can also divide a physical device into multiple virtual devices for use by multiple virtual machines. However, SR-IOV technology is primarily used for network devices to achieve efficient network virtualization. Compared to the sdev technology in GPU virtualization, SR-IOV places more emphasis on I/O virtualization rather than GPU compute virtualization. Therefore, the application scenarios for the two differ.

SR-IOV technology requires hardware support, so it can only be used when the hardware platform supports it. SR-IOV devices implement virtualization using two types of functions: VF (Virtual Function) and PF (Physical Function). The PF is the actual physical device, while VFs are the multiple devices virtualized through SR-IOV technology. VFs are independent of one another; they can directly access the PF and the physical network to achieve efficient data transmission.

When attempting to migrate a virtual machine that uses SR-IOV virtualized GPU devices to another physical machine, difficulties may arise. This is because the VFs used by SR-IOV technology are provided only by a single physical device (PF), and VFs cannot be directly migrated from one physical device to another. Additionally, when using SR-IOV technology, the mapping relationship between the virtual machine and the VF is predefined during the virtualization process, which limits flexibility during migration.

Therefore, if migration operations are required for virtual machines using SR-IOV technology, extra work may be needed. In the case of SR-IOV, it appears that the VF on the GPU hardware is accessed directly, and live migration essentially involves disconnecting and reconnecting. For example, you might pre-configure the same VF mapping relationships on the target physical machine as on the source physical machine, and ensure that the correspondence between VFs and physical devices is maintained during the migration process, so that the virtual machine can correctly use GPU resources after migration.

You can refer to the following chart showing potential latency when migrating using SR-IOV:

file

In conclusion, virtualization technology offers many benefits but also presents challenges, particularly in high-performance computing scenarios. To make virtualization technology more flexible and usable, continuous advancement and innovation are required at both the technical and engineering practice levels.

Final Form Chosen: mdev

mdev (Mediated Device) is a software virtualization technology that enables the virtualization of a physical device into multiple virtual devices for use by multiple virtual machines. Similar to SR-IOV technology, mdev requires hardware platform support; however, it does not require the physical device to directly support SR-IOV.

In terms of implementation, mdev achieves virtualization of physical devices by constructing virtual device drivers within the host kernel. These virtual drivers in the host kernel allow multiple virtual machines to access the physical device directly without conflict or interference. Compared to SR-IOV technology, mdev offers a more flexible mapping relationship between virtual devices, which makes it easier to migrate virtual machines from one physical machine to another.

NVIDIA vGPU technology utilizes mdev technology at the underlying level to implement GPU resource virtualization and allocation. Therefore, it is very easy to migrate vGPUs between physical hardware.

(It is worth noting that, as shown in the figure, this diagram gives the impression that migration between devices is very straightforward.)

file

mdev technology is not limited to GPU devices; it can also be used for other types of PCIe devices, such as Network Interface Cards (NICs). mdev is a general-purpose device virtualization framework designed to provide hardware resource partitioning and virtualization capabilities for various types of devices. For devices that support mdev, their drivers can leverage the mdev framework to implement resource virtualization and allocation, enabling multiple virtual machines or containers to share the resources of the physical device. mdev technology is also highly useful in network virtualization scenarios, such as in data centers and cloud computing environments. By partitioning the resources of a physical Network Interface Card into multiple virtual devices, it can reduce resource consumption, improve resource utilization, and simplify network management. In summary, mdev technology is applicable to a wide range of PCIe devices, including but not limited to GPUs and Network Interface Cards. Device drivers can utilize the mdev framework to provide hardware resource virtualization and sharing capabilities for virtualized environments.

Purchasing and Inserting Compute Cards

Purchase

I purchased the Nvidia P40. Before starting, I tried using the P4. The price of the P40 is approximately 1,100 CNY per card. In contrast, the P4 is very cheap, costing only about 400 CNY. It is recommended to purchase from Xianyu.

file

Before purchasing, you must consider: What you need might not be a single compute card! Purchasing only one compute card results in only one host having vGPU functionality, while your vGPU still cannot benefit from migration capabilities! This means you will still lose capabilities such as VMware’s DRS (Distributed Resource Scheduling), DPM (Dynamic Power Management), and HA (High Availability)! I still recommend symmetric compute, which means that for a cluster, if there are N hosts, you should purchase N compute cards!

Heat Dissipation

Warning! The P40 does not have a cooling fan!!! Do not install it in consumer devices!!! This is because the P40 relies heavily on server airflow for cooling!

file

For example, the server I use is a PowerEdge R730 Rack Server. It perfectly directs the machine's airflow toward the P40.

file

If you plan to install the compute card in a consumer device, you must 3D print your own fan!

Request a License

Apply for an NVIDIA Trial

To perform CUDA computation or display output on an Nvidia VGPU, you must install the NVIDIA Grid driver. However, this driver is paid, and it is very difficult for individuals to purchase it.

Note: Merely obtaining the binary is not enough. After installing the driver, compute cards require license renewal at regular intervals. It functions more like a KMS activation with an extremely short lifecycle. Once the activation server determines that your authorization has expired, it will immediately lock the compute card.

file

To apply for an evaluation, open the NVIDIA Vgpu Evaluation Center, select "Register For Trial," fill in your email address according to the prompts below, and provide other information truthfully.

file

Here, fill in your company's information in detail. Note: You must provide information for a real company! Nvidia strictly prohibits individual users from applying for vGPU Licenses! Do not use personal email addresses!

file

After submitting the application, wait for a period of time (from a few minutes to a few days). If everything goes smoothly, you will receive a confirmation email.

file

Create License Server

Go to the NVIDIA Dashboard: https://nvid.nvidia.com/dashboard/#/dashboard. Log in using the email address you used when making your application.

Open: NVIDIA LICENSING PORTAL.

  • Select License Server, and click Create Server
  • Fill in the name and description, check Express CLS Installation, and select Next

file

Assign licenses to the server. I assigned 16 vWs licenses and 16 vApp licenses to the license server. You can adjust the quantities according to your deployment scale.

file

Once completed, click Next, and then Create Server.

On the left side, select Server Instances. Then, in Actions, select Generate Client Configuration Token. Select the license server you just created, and then click Download. This file will be used later when configuring licenses for the server.

file

Install Drivers for the Host

Download Drivers

First, download the driver from the NVIDIA official website. Filter by the latest major Product Version, such as 15. Be sure to select the version compatible with your Vsphere. I am still using Vsphere 7.0, so I am using this driver here.

file

Note: This driver is very large. It may be over 2GB. Note: Do not upload this driver to the internet for everyone to download!

file

Here, you will see these two compressed files. Both are required!

NVD-VGPU-702_525.105.14-1OEM.702.0.0.17630552_21514677.zip
nvd-gpu-mgmt-daemon_525.105.14-0.0.0000_21514245.zip

Upload the Driver to VMware Lifecycle Manager

I recommend using VMware Lifecycle Manager to install drivers. Open VCenter Server, navigate to Lifecycle Manager, and review the existing driver configurations. Filter by the name Nvidia:

file

As shown in the image, VMware Lifecycle Manager has already identified the driver 525.105.14. If the latest version of the driver is not present here, you need to upload it.

Select Actions -> Import Updates to upload the driver.

file

Generally, after uploading the entire zip file, you may need to wait for several minutes for VCenter to sync the updates. If there are any issues, you can try uploading the metadata.zip file included within.

Once the upload is complete, modify the Image settings of the compute cluster to add the Grid driver.

Afterwards, you need to reinstall the system on all hosts. If your Vsphere configuration is correct, this will not cause downtime. You can watch the video I recorded:

(If the video does not play, please click here)

<video style="width:80%" src="https://videos.aiursoft.cn/media/encoded/7/anduin/8ae31546ab3c42759840a394d6ad11f8.8ae31546ab3c42759840a394d6ad11f8.v.mp4.mp4"></video>

After the installation is complete, go to the configuration section of the host and select Graphics to view the compute card.

file

Here, you will see the following options:

file

Be sure to select the latter, shared direct.

The former is similar to the semi-virtualized graphics acceleration used in VM Workstation, whereas the latter corresponds to GRID/MxGPU.

file

Reference documentation: https://techzone.vmware.com/resource/deploying-hardware-accelerated-graphics-vmware-horizon-7#comparison-of-the-types-of-graphics-acceleration

Assigning vGPU to a Virtual Machine

Note: Before starting, it is recommended to disable the DRS and DPM features in the Cluster! Enabling vGPU will temporarily prevent the virtual machine from being migrated. However, vCenter may still attempt to migrate this virtual machine, or DPM may shut down the host, only to encounter an error, resulting in an infinite deadlock.

Deciding Which vGPU to Assign to the Virtual Machine

First, migrate the virtual machine to a host with GPU resources. If all hosts in your entire cluster have GPUs, this step is not required.

In vCenter Server, add a virtual PCI device to your virtual machine. Select passthrough for NVIDIA GRID vGPU.

file

When allocating a virtual GPU to the virtual machine, you can choose from different vGPU types.

Since we need to use OpenCL/CUDA computing capabilities, you must ensure that the GPU type you select supports CUDA operations. Generally, all Q-Series and C-Series vGPUs support CUDA computation (Tesla M6, Tesla M10, and Tesla M60 only support 8Q), while A-Series and B-Series do not support CUDA computation. Please ensure you select a Q/C Series vGPU. It is recommended to choose Q-Series, as it offers the most powerful capabilities. For details, please refer to the Nvidia User Manual.

Install Drivers for the Virtual Machine

Ensure the Compute Card Is Detected

Before starting the driver installation, ensure that the virtual machine can recognize the compute card:

lspci -nn | grep -Ei "3d|display|vga"
lshw -C display

Find your calculation card in the output. For example:

file

Install GRID Drivers for Your Virtual Machine

In the previous steps, we have already downloaded the drivers.

Generally, the client driver version should match the hypervisor driver version. There are three distribution formats for the drivers: deb, rpm, and runfile. If your platform supports installing the corresponding software packages, it is not recommended to install via runfile.

Here, we use Ubuntu 22.04 LTS as an example. After downloading the corresponding version of the drivers to the server, you can complete the installation using the following command:

sudo apt install ./nvidia-linux-grid-525_525.105.17_amd64.deb

file

Here, I used apt install to complete the installation instead of dpkg -i, because apt can automatically install missing dependencies.

After the installation is complete, enter nvidia-smi. At this point, you should be able to see the vGPU information.

file

Configuring the GRID Driver License

First, edit the gridd configuration file:

sudo vim /etc/nvidia/gridd.conf

Find FeatureType and change it to 1. Keep all other options unchanged.

Next, navigate to /etc/nvidia/ClientConfigToken (create it if it does not exist), insert the previously downloaded Client Configuration Token into it, and then use chmod 744 to modify its permissions.

Once done, restart nvidia-gridd:

sudo systemctl restart nvidia-gridd.service

Finally, use nvidia-smi -q | grep License to confirm that the license has been installed correctly. If it is installed correctly, you should see output similar to the following:

vGPU Software Licensed Product
        Product Name                      : NVIDIA RTX Virtual Workstation
        License Status                    : Licensed (Expiry: 2023-4-20 12:14:21 GMT)

file

Install CUDA

Note: When selecting a CUDA version, keep in mind that Grid drivers typically lag behind standard NVIDIA drivers by several versions. Therefore, you may not be able to install the latest version of CUDA. You can check the driver version requirements for different CUDA versions in the CUDA release notes.

Note: On any Linux system, regardless of the distribution, you must not use platform packages (deb, rpm, etc.) to install the cuda-toolkit. You must use the runfile installation method! If you install using platform packages, the cuda-driver will conflict with the installed grid-driver during installation, preventing the installation from proceeding!

Note: On any Linux system, regardless of the distribution, do not attempt to install it via package managers either. sudo apt install nvidia-cuda-toolkit. This command will remove the Grid driver!

Check the CUDA requirements for NVIDIA drivers here: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

file

Check VGPU support for CUDA here: https://docs.nvidia.com/cuda/vGPU/index.html

file

Note: Some AI software also has CUDA version requirements. You should determine the appropriate CUDA version to install based on its documentation.

Open the cuda-toolkit-archive, select the version you want to install, and then choose your corresponding system to obtain the download link for the corresponding CUDA toolkit.

file

Note: The file cuda-repo-ubuntu2204-12-1-local_12.1.1-530.30.02-1_amd64.deb may be quite large (approximately 3GB).

After downloading the runfile, run it using sudo.

file

On the first license agreement page, type accept:

file

Then, on the component selection screen, uncheck the installation of cuda-driver; otherwise, the grid driver will be overwritten!

file

After the installation is complete, edit the system PATH environment variable using sudo vim /etc/environment.

Append :/usr/local/cuda/bin to the end of PATH (modify the version number as needed).

Restart your shell, and then use the following command to verify that CUDA has been successfully installed:

nvcc --version

Finally, considering the need for large-scale deployment, I went ahead and wrote a handy script for my own use:

# 我把驱动、CUDA和License都挂载到了 /mnt/Shared/nv/
sudo lspci | grep NVIDIA
cd /mnt/Shared/nv/
sudo apt install ./nvidia-linux-grid-525_525.105.17_amd64.deb -y
sudo nvidia-smi
sudo rm /etc/nvidia/ClientConfigToken/client_configuration_token*
sudo cp ./client_configuration_token_22-04-2023-15_14_46.tok /etc/nvidia/ClientConfigToken
sudo chmod 744 /etc/nvidia/ClientConfigToken/client_configuration_token_22-04-2023-15_14_46.tok

sudo systemctl restart nvidia-gridd.service
sleep 10
sudo nvidia-smi -q | grep License

sudo ./cuda_12.1.1_530.30.02_linux.run

PATH="$PATH:/usr/local/cuda-12.1/bin"
echo "PATH=\"$PATH\"" | sudo tee /etc/environment > /dev/null
source /etc/environment

nvcc --version
nvidia-smi

Ensure the Virtual Machine Can Migrate Across Hosts

Even after GRID and CUDA are fully configured, the virtual machine can still be smoothly migrated between hosts.

file

Divide SKUs

Note: Once your GPU is allocated to a 3q (3 GB VRAM VM), it can no longer be allocated to other types of vGPUs.

For example: For a P40 with a total of 24 GB of VRAM, you can allocate 3 VMs with 8 GB of VRAM each. However, once it is partitioned into 8 GB segments, it cannot be further allocated to machines with 4 GB or 2 GB of VRAM!

Therefore, you must plan in advance how to partition each physical GPU. Here, I recommend designing two partitioning schemes:

  • 1/8 GPU, where each vGPU has 3 GB of VRAM. (For simple gaming, encoding/decoding, and computation)
  • 1/3 GPU, where each vGPU has 8 GB of VRAM. (For running larger models)

Test Migration

First, enable migration in VCenter Server:

file

Enable the configuration option: vgpu.hotmigrate.enabled.

file

Additionally, pay attention to consistent host settings! If you have enabled ECC, all hosts must have ECC enabled!

Then try the migration! Note: vGPU migration cannot cross hosts with different partitioning types!

For example: If your GPU is partitioned into 1/8, with each vGPU having 3GB of VRAM, you cannot migrate a 3GB VRAM vGPU to a GPU partitioned into 1/3.

Start Alchemy

You've got it all figured out. Start refining the elixir!

Installing cuDNN

For many applications, cuDNN is required for model training.

Let's install cuDNN first! Here is a detailed guide on how to install cuDNN on Ubuntu 22.04: https://anduin.aiursoft.cn/post/2023/4/24/how-to-install-cuda-and-cudnn-on-ubuntu-2204-and-test-if-its-installed-successfully.

Video Encoding

After the driver installation is complete, you can already use the GPU for hardware-accelerated encoding.

You can directly use ffmpeg for encoding. Select the encoder: h264_nevnc, and try to see if it can encode successfully.

file

While encoding, you can check the GPU's compute resource usage.

Generally, video encoding and decoding only requires a few hundred MB of VRAM.

file

If ffmpeg can encode and decode normally, then the media applications you are using are likely able to enable hardware acceleration as well.

file

During hardware-accelerated encoding and decoding, you can also see the GPU being utilized:

file

Running the AI Application

Here are a few AI projects that can run locally on a P40!

There are some common libraries that your AI application might need:

sudo apt install python3
sudo apt install libgl-dev

You can run Stable Diffusion WebUI directly after cloning it:

file

When using Stable Diffusion, you can see Python utilizing the GPU and the VRAM usage in nvidia-smi.

file

Monitoring Temperature - If You Don't Want Noise

Server fans typically generate significant noise to offset the acoustic footprint of CPUs and PCIe devices. However, if your server is located in a home environment, this can become a major nuisance.

You need an effective method to retrieve the server's temperature, calculate the appropriate fan speed, and then write a program to control the fan speed.

To obtain the fan speed, run the following command on the host machine (Note: not on the vGPU):

nvidia-smi -q -d temperature

file

If something goes wrong……

Data centers with GPU computing power are significantly more difficult to manage than those with CPU-only computing power.

If issues arise, first check the following:

  • Has the vGPU license been successfully activated? nvidia-smi -q

Then, if there have been recent changes, check the following items:

  • The Linux kernel must support the Grid Driver
  • The Host Driver must match the Guest Driver
  • The Guest Driver must be consistent with (or support the version of) CUDA
  • CUDA must be consistent with the AI application to be run

Additionally, check:

  • In the host's BIOS settings, ensure the following are enabled:
    • IO MMU
    • PCIE AER
    • SR IOV (Although our graphics cards do not use SR IOV, I still recommend enabling this option)

file

Don't forget to check:

  • In VCenter, can the card be detected, and is it configured as Shared Direct Vendor shared passthrough graphics?
  • Are the physical GPU settings consistent (e.g., ECC on/off)?