---
title: "Provision of Virtual Machines with GPU acceleration"
canonical: "https://www.virtcloudrocks.com/space/VSPHTECH01/852163/Provision%20of%20Virtual%20Machines%20with%20GPU%20acceleration"
format: markdown
---
Scope The main scope of this technical article is to describe in a comprehensive way how we can provision virtual servers (VMs) with GPU acceleration; those VMs are to be used to perform Deep Learning experiments which require high processing power, far beyond the conventional CPUs. To achieve this we performed below, we virtualized, installed vSphere ESXi, a set of two Dell C4130 servers, each of one equipped with 2 X NVIDIA P100 GPU cards, then we created a new virtual cluster which included the two GPU-enabled virtualized ESXi servers, and finally,  we created some VMs and presented to those VMs the GPUs through Direct Pass-through I/O mode. Technologies When it comes to use GPUs with virtual servers that will be used for HPC, Machine / Deep Learning purposes, there are multiple ways that this can be achieved,  https://blogs.vmware.com/apps/2018/07/using-gpus-with-virtual-machines-on-vsphere-part-1-overview.html Summarizing below are the possible options, when it comes provision of VMs with GPU capabilities. Use DirectPath I/O (Passthrough) mechanism for GPUs Through setting up the NVIDIA GRID technology for GPUs Through setup for the BitFusion Flexdirect method of using GPUs In our case we shall go with the first option, i.e. by using DirectPath I/O (Passthrough) mechanism for presenting underlying GPUs to virtual machines. The below diagram depicts a high level view of the architecture, Some further notes to have in mind, (quoted from  VMware blog ), The main reasons for using the passthrough approach to exposing GPUs on vSphere are: you are taking your first steps to exposing GPUs in virtual machines so as to move the end users away from storing their data and executing workloads on physical workstations;  there is no need for sharing the GPU among different VMs, because a single application will consume one or more full GPUs you need to replicate a public cloud instance of an application, but using a private cloud setup. An important point to note is that the passthrough option for GPUs works without any third-party software driver being loaded into the ESXi hypervisor. When using passthrough mode, each GPU device is dedicated to that VM and there is no sharing of GPUs amongst the VMs on a server. The vSphere features of vMotion, Distributed Resource Scheduling (DRS) and Snapshots are not allowed with this form of using GPUs with a virtual machine Hardware Specifications Below table shows the basic configuration of the servers that we will use in the context of that project. Currently we have available 2 servers, each one having below hardware specifications. Server Model Dell PowerEdge C4130 Processor 2 x Intel Xeon E5-2650 v4 2.2GHz,30M Cache,9.60GT/s QPI,Turbo,HT,12C/24T (105W) Max Mem 2400MHz Memory 256GB, RDIMM, 2400MT/s, Dual Rank, x8 Data Width Local Storage 2 x 960GB Solid State Drive uSATA Mix Use Slim MLC 6Gbps 1.8in Hot-plug Drive, SM863 GPUs 2 x NVIDIA Tesla P100 16GB Passive GPU Detailed specifications for the GPU card can be found to the below link,  http://images.nvidia.com/content/pdf/tesla/NV-tesla-p100-pcie-PB-08248-001-v01.pdf Network (1G) 2 x 1GbE, (onboard network card) Network (10G) 2 x 10GbE, (Intel Ethernet X540 DP 10GBASE-T Server Adapter, Low Profile) PERC RAID Controllers S130 (software RAID) Firmware upgrade On both servers we performed below firmware upgrades, in order to make sure that both servers have the latest firmware upgrades, especially BIOS. iDRAC Lifecycle controller BIOS Intel Network Cards Servers Preparation Virtual Disk Configuration Each of the servers has a PERC S130 RAID controller, which is software-based RAID controllers and for the time being Dell doesn't provide drivers for ESXi hypervisor, thus even if we create a virtual disk eventually during the hypervisor installation the new virtual disk won't be visible. Thus in our case, for those specific servers, make sure that change  BIOS settings  of the server →  SATA settings  and select the option  AHCI Mode  for the  Embedded SATA In case that the server has a hardware RAID controller, e.g. PERC H330/H810 then proceed and create a virtual disk in RAID-1 mode, in order to have availability in case of OS disk failure. The virtual disk has to be created before the installation of the hypervisor . vSphere Host Preparation In this section we describe all the necessary steps that need to be done in order to prepare the host to support the underlying GPUs cards in Passthrough mode. In a high level view, below steps should be followed in order to prepare successfully a vSphere host with GPU(s) in Passthrough mode. Check that GPU is supported from the vendor to be configured in Passthrough mode. Check whether your PCI GPU device maps memory regions whose size in total is more than 16GB.  The NVIDIA Tesla P100 is an example of such a GPU device, so this is applicable in our case, where each server is equipped with two NVDIA Tesla P100 GPU cards. Post installation and initial configuration of the ESXi, you need to configure the PCI devices availability to the server. Server BIOS - Memory mapped I/O above 4GB In server BIOS, 4G mapping/encoding should be enabled. For the Dell C4130 server, the setting can be found under  BIOS settings  (F2) →  System BIOS  →  Integrated Devices,  https://qrl.dell.com/Files/en-us/Html/Manuals/C4130/Integrated%20Devices%20screen%20=GUID-4245C3AF-F834-4B25-A4E2-2297612C9073=1=en-us=.html Important note If your GPU card does NOT need PCI MMIO regions that are larger than 16GB, then you may skip this section (“Host BIOS Setting”). vSphere Installation / Configuration This section provides some details related with installation and configuration of the vSphere ESXi to the server. We shall provide the basic steps for installation and configuration and not all detailed steps that have to be followed. VMware Hardware Compatibility List Below it can be seen the hardware compatibility of the server with the vSphere 6.5U2 https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server&productid=38846&vcl=true For the vSphere ESXi installation we used a Dell customized ISO with the ESXi,  https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=ckc15 ESXi Installation In that section we shall not describe in detail all the steps followed to install and configured the vSphere ESXi hypervisor. Mainly, we focus on an issue that we faced towards the installation process of the hypervisor, and a workaround that was applied to overcome the issue faced. More specifically, in a point of time, after selection of the disk to which the hypervisor will be installed, we faced an error, (see below screen), To overcome this error, we refer to article,  https://definitelyalive.com/2017/03/18/fixing-cant-have-a-partition-outside-the-disk/ Below it can be seen details of all the steps that we followed in order to resolve that error and proceed further with the installation of the vSphere ESXi to the server. Step Task 1 Reboot the server, and re-initiate the ESXi hypervisor installation 2 When you reach at the step where the storage device for installation needs to be selected, press  Alt+F1 By pressing F1, at that stage, you can see details of the disk selected 3 Login using root and no password 4 Move to /dev/disks and list the contents $ cd /dev/disks 
$ ls -la 5 Run below command to convert the disk to GPT, $ partedUtil mklabel <name of the disk> gpt 6 Press Alt+F2 in order to return back to list of storage devices, select the 1st disk, (the one that we converted to gpt), and proceed with ESXi installation vCenter Configuration As soon as the installation and initial configuration of the hypervisor finish, you need to connect to the vCenter server, (in our case, the one that handles the Compute/Payload clusters), and do some further configuration. Below summarized list of actions need to be done, Create a new virtual cluster Add the host as member of the new cluster Proceed and configured vDS and networking configuration Present datastores to the hypervisor that will be used to store virtual servers files If the cluster will participate in VXLAN, prepare the cluster for NSX / VXLAN networking Virtual Cluster A new virtual cluster has been configured on one of the vCenter servers which handles the Compute Clusters of our environment; we named the new cluster as  GPU-Cluster-I In passthrough mode, each GPU device is dedicated to that VM and there is no sharing of GPUs amongst the VMs on a server. The vSphere features of vMotion, Distributed Resource Scheduling (DRS) and Snapshots are not allowed with this form of using GPUs with a virtual machine, thus neither HA or DRS have been enabled to that cluster. The new hypervisors were added as new members to the new virtual cluster. vDS / Networking Configuration Both servers are added as members of the proper portgroups in Management vDS, and two vmkernels have to be created in the proper dPGs, one for vMotion and one NFS. Furthermore, the servers should be added to Production vDS, which have dPGs related with other Production data, e.g. NFS portgroups, etc. Datastores Mounted the proper LUNs / datastores at the new hypervisors where VMs will be deployed to. NSX/VXLAN Cluster Preparation This section is applicable in case that you have installed and configured NSX and VXLAN overlay Logical switches to which the VMs are deployed to. Prepared the new virtual cluster for NSX and configured VXLAN to the new server. Whenever a new server is added to the new cluster, it will be automatically installed the NSX VIBs and the new server will be ready for VXLAN. In case that the new cluster will be prepared for VXLAN, do not forget to update the Transport zone to include this cluster. Initialize the PCI GPU devices to the hosts An installed PCI-compatible GPU hardware device is initially recognized by the vSphere hypervisor at server boot-up time without having any specific drivers installed into the hypervisor. Below steps have to be followed in order to make the PCI installed GPU devices available to the vSphere hypervisor. Step Task 1 Go to vCenter server that has the new cluster of server → click on the cluster  GPU-Cluster-I 2 Click on one of the hosts. 3 Click on  Configure 4 Scroll down to the hardware section → click  PCI Devices 5 Click  Edit 6 Depending on how many GPU cards has the server, scroll down and select the GPU cards, one by one, 7 Reboot the server. 8 After server rebooted you should be able to see the GPU cards under the PCI devices; each GPU card that will be configured in Passthrough mode has to be in status  Available , (as per below figure). 9 Repeat previous steps for the other server. Virtual Machine Configuration As soon as the GPU card is visible as a DirectPath I/O device on the host server, we then proceed to the configuration steps for the virtual machine that will use the GPU. Configure EFI or UEFI boot options for the virtual server Adjusting the Memory Mapped I/O Settings for the VM Configuring EFI or UEFI Mode The new virtual machine will be created in the vSphere Client in the normal way, but there is one part of this process that needs special attention, and it has to do with the boot options for the VM.  More specifically, prior the OS installation to the virtual machine we need to make sure that the guest operating system within the virtual machine must boot in EFI or UEFI mode for correct GPU use . Below it can be seen how you can enable the UEFI or EFI boot option for the virtual machine. Important note If your GPU card does NOT need PCI MMIO regions that are larger than 16GB, then you may skip this section (“Configuring EFI or UEFI Mode”). To configure the setting for this, select the virtual machine in the vSphere Client and use the following menu items,  Edit Settings  →  VM Options  →  Boot Options,  in order to get to the  Firmware  parameter. Ensure that the  UEFI  or  EFI  is enabled in the  Firmware  area as it can be seen below, Adjust Memory Mapped I/O to the VM The PCI GPU device that we are using maps memory regions of more than 16GB in size, so we will need to set two specific configuration parameters for the VM in the vSphere Client, as described below. If your GPU card does NOT need PCI MMIO regions that are larger than 16GB, then you may skip this section (“Adjusting the Memory Mapped I/O Settings for the VM”). For a GPU device that maps more than 16GB of memory, proceed as follows. Step Task 1 In the vSphere Client, select the virtual machine and then  Edit Settings  →  VM Options → Advanced → Configuration Parameters → Edit Configuration 2 Add the following two parameters, with the value of the second parameter, as shown below, The value of the second parameter is adjusted to suit your specific GPU requirements. Below it can be seen an easy way to perform the calculation of the value. Count the number of high-end PCI GPU devices that you intend to pass into this VM. This can be one or more GPUs. Multiply that number by 16 and round it up to the next power of two. For example, to use passthrough mode with two GPU devices in one VM, the value would be: 2 * 16 = 32, rounded up to the next power of two to give 64. Install the Guest OS and GPU drivers As soon as you finalize the configurations described in the previous sections, (if those are needed), proceed and install the Guest Operating System into the virtual machine. As we mentioned befire this should be an EFI or UEFI-capable operating system, if your GPU card has a requirement for large PCI MMIO regions. Furthermore, the standard vendor GPU driver must also be installed within the guest operating system. In our case since our GPU card is the NVIDIA Tesla P100, we need to install the NVIDIA Cuda framework, in order to make sure that the GPU will be full operational from the VM. Although this is out of the scope of the article, for completeness purposes we shall describe installation of the NVIDIA Cuda on Linux. NVDIA Minimal Installation on Ubuntu 16.04 In case that you need to perform a minimal installation of the CUDA packages with no GUI, etc., please follow below steps, 1 - Download packages lists from the repositories and updates $ sudo apt update 2 - Fully upgrade the system packages $ sudo apt dist-upgrade -y 3 - Run below command in to get linux headers, build tools and dkms for automatic rebuilding of nvidia stuff with new kernels $ sudo apt install -y linux-headers-generic-hwe-16.04 build-essential dkms 4 - Blacklist nouveau $ echo "blacklist nouveau" > /etc/modprobe.d/blacklist-nouveau.conf
$ echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist-nouveau.conf 5 - Update initramfs $ sudo update-initramfs -u 6 - Download CUDA Toolkit,  https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux , and run below command to proceed with installation $ sudo ./cuda_10.0.130_410.48_linux.run --silent --driver --toolkit --samples --samplepath=/usr/cuda-10.0/samples 7 - Verify correct operation of the GPU cards installed and configured with the system, (see Section, Validation and Testing). NVIDIA Cuda Full Installation on Ubuntu 16.04 For the purposes of our project the OS installed to the virtual server is Ubuntu 16.04.05 LTS x64. As soon as the OS installation and initial configuration finishes to proceed with the NVIDIA Cuda framework installation you need to follow below set of steps, Pre-reqs To install CUDA on your system, you will need the following installed: A supported version of Linux with a gcc compiler and toolchain Download the NVIDIA CUDA Toolkit (available at  http://developer.nvidia.com/cuda-downloads ); the package that we used for Ubuntu 16.04 OS can be downloaded from this link  https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda-repo-ubuntu1604-10-0-local-10.0.130-410.48_1.0-1_amd64 Pre-Installation tasks Some actions must be taken before the CUDA Toolkit and Driver can be installed on Linux: Verify the system is running a supported version of Linux. Verify the system has gcc installed. Verify the system has the correct kernel headers and development packages installed. Download the NVIDIA CUDA Toolkit. Handle conflicting installation methods. Verify supported OS version To verify supported version of the OS type the following at the command line, $ uname -m && cat /etc/*release Verify gcc installed To verify thar gcc compiler is installed, type the following command in a terminal, $ gcc --version Verify correct kernels and development packages are installed To get the version of the kernel installed to the OS, you can type below command in a terminal, $ uname -a In case that you need to install correct version of the OS kernel and development packages, you may run below command, $ sudo apt-get install linux-headers-$(uname -r) Download the NVIDIA Cuda toolkit Download the NVIDIA CUDA Toolkit (available at  http://developer.nvidia.com/cuda-downloads ); the package that we used for Ubuntu 16.04 OS can be downloaded from this link  https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda-repo-ubuntu1604-10-0-local-10.0.130-410.48_1.0-1_amd64 Handling previous CUDA installations Before proceed with CUDA framework installation, you need to clear any previous installations of CUDA. To do this you can see this link  https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#handle-uninstallation Installation process on Ubuntu 1 - Move to the folder where you downloaded the installation file and run below command, $ dpkg -i cuda-repo-ubuntu1604-10-0-local-10.0.130-410.48_1.0-1_amd64 2 - Install the CUDA public key $ sudo apt-key add /var/cuda-repo-10-0-local-10.0.130-410.48/7fa2af80.pub 3 - Run below commands to finalize the installation of the CUDA framework. Please note that installation might take some hour to finalize. $ sudo apt-get update
$ sudo apt-get install cuda 4 - When the installation will finish, please proceed and reboot the server,. $ sudo reboot Post-Installation steps After the successful installation of the CUDA framework you need to do some post-installation mandatory configurations on the server. Environment PATH setup The PATH variable needs to include /usr/local/cuda-10.0/bin To add this path to the PATH variable, execute below command on the command line, $ export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}} Furthermore, update the environmental variables for the x64 OS, so run below command on the command line $ export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} POWER9 Setup Make sure that the NVIDIA Persistence Daemon is configured to be automatically started for POWER9 installations. Check the status of the service by running below command on a terminal, $ systemctl status nvidia-persistenced Validation of CUDA installation There are some further tasks that are recommended to be done, as soon as the installation of CUDA framework finishes that will validate correct installation and configuration of the CUDA. More details for some recommended tasks can be found here,  https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#recommended-post Install Persistence Daemon NVIDIA is providing a user-space daemon on Linux to support persistence of driver state across CUDA job runs. The daemon approach provides a more elegant and robust solution to this problem than persistence mode.  The NVIDIA Persistence Daemon can be started as the root user by running the below command on a terminal, $ /usr/bin/nvidia-persistenced --verbose This command should be run automatically on boot. Install Writable Samples CUDA installation comes with a set of samples which can be used to test, benchmark, verify correct operation of the CUDA framework, etc. In order to modify, compile, and run the samples, the samples must be installed with write permissions. A convenience installation script is provided that will install the CUDA samples; run below command. $ cuda-install-samples-10.0.sh cuda-samples Before being able to use the CUDA samples you need to compile all the files, thus to do so run below commands, $ cd cuda-samples/NVIDIA_CUDA-10.0_Samples/
$ make More details, if needed regarding building the CUDA samples can be found here,  https://docs.nvidia.com/cuda/cuda-samples/index.html#building-samples Assigning a GPU device to the virtual machine At this point the OS and CUDA framework installation has finished, and we need to proceed and assign the GPU or GPUs to the virtual machine that we created. To enable the virtual machine to have access to a PCI device(s), we need to follow below steps, 1 - Connect to vSphere web client 2 - Power off the virtual machine 3 - Select the virtual machine →  Edit Settings 4 - Scroll down and go to  New Device , select  PCI device , → press  Add Memory Reservation When a PCI device is assigned to a VM, the virtual machine must be configured to have a memory reservation for the full configured memory size for the VM. This can be done in the vSphere client by changing settings of the VM, i.e. select the VM → Edit Settings → Virtual Hardware →  Memory to access and change the value into the Reservation area. Alternatively, during the addition of the new PCI device, you shall see a warning message related to the memory reservation needed and you can press the button  Reserve all memory As soon as you perform this task, you may power on the server and you are ready to use the VM with GPU acceleration. Validation and Testing Verify the GPU card To verify that the GPU card is CUDA-capable, connect to the server (VM) and run below command through a terminal, $ lspci | grep -i nvidia Get details of the GPU(s) installed You can run below command in a console in order to get some basic details of the GPU or GPUs installed to the system, e.g. VBIOS $ sudo nvidia-smi --query-gpu=gpu_name,gpu_bus_id,vbios_version --format=csv Query GPU device state On a command line run below command in privileged mode, to query the state of the GPU device. $ sudo nvidia-smi Through nvidia-smi you can get numerous details, metrics, etc., related with all GPU devices that are installed yo your system. You can find further details here,  https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries Run DeviceQuery After compilation of the CUDA samples, find and run deviceQuery under NVIDIA_CUDA-10.0_Samples. To do so run below commands on a terminal, $ cd cuda-samples/NVIDIA_CUDA-10.0_Samples/bin/x86_64/linux/release
$ ./deviceQuery If the CUDA software is installed and configured correctly, the output for deviceQuery should look similar to below figure, and the result should be PASS. Bandwidth Test Running the bandwidthTest program we can verify and ensure that the system (virtual machine) and the CUDA device are able to communicate correctly; the output can be shown to below figure and result should be PASS. Assign multiple GPU cards in a virtual machine In case that the server is equipped with more than one GPU devices, we have the option to present more than one GPU devices to the virtual server. In order to to do this we need to perform below actions, Adjust the Memory Mapped I/O Settings for the VM, since we are modifying the number of GPU devices that will be presented to the VM Add the extra GPU devices to the VM Adjust  Memory Mapped I/O Settings for the VM to support more than one GPU In our case our server has two NVIDIA Tesla P100 GPU devices and we shall perform the necessary adjustments to Memory Mapped I/O. Further details how this can be done can be found in a previous section of this article,  AdjustMemory-Mapped-I/O-to-the-VM In our use case, we shall re-configure the VM settings, and set the parameter pciPassthru.64bitMMIOSizeGB = 64 Add the new GPU to the VM Following steps followed to previous sections, we need to proceed and add the second GPU card to the VM. After the successful configuration the VM settings should look as per below figure, Then we can proceed and power-on the server and verify that both GPUs are visible to the server. Bench-marking the GPUs  In order to stress test the GPu cards and verify correct operation, I have found a nice article here,  http://wili.cc/blog/gpu-burn.html You can download from the GitHub repo the files and do a CUDA compilation, and then run it to stress test the cards. Below I run the program for 5 mins using double-precision, and below some screenshots towards the execution, In order for the complication to succeed, I had to perform below steps, 1 - Install NVIDIA Cuda Toolkit $ sudo apt install nvidia-cuda-toolkit 2- Update the Makefile with proper locations of the NVCC and gcc CUDAPATH=/usr/local/cuda
# Have this point to an old enough gcc (for nvcc)

GCCPATH=/usr/bin/gcc
NVCC=/usr/local/cuda/bin/nvcc

CCPATH=${GCCPATH}/bin
drv:
 PATH=${PATH}:.:${CCPATH}:${PATH} ${NVCC} -I${CUDAPATH}/include -arch=compute_30 -ptx compare.cu -o compare.ptx
 g++ -O3 -Wno-unused-result -I${CUDAPATH}/include -c gpu_burn-drv.cpp
 g++ -o gpu_burn gpu_burn-drv.o -O3 -lcuda -L${CUDAPATH}/lib64 -L${CUDAPATH}/lib -Wl,-rpath=${CUDAPATH}/lib64 -Wl,-rpath=${CUDAPATH}/lib -lcublas -lcudart -o gpu_burn Tables of Contents VMware articles on HPC / ML Using GPUs with Virtual Machines on vSphere – Part 1: Overview Using GPUs with Virtual Machines on vSphere – Part 2: VMDirectPath I/O Using GPUs with Virtual Machines on vSphere – Part 3: Installing the NVIDIA GRID Technology Using GPUs with Virtual Machines on vSphere – Part 4: Working with BitFusion FlexDirect Enabling MLaaS with GPU acceleration on top of VMware vSphere How to Enable Compute Accelerators on vSphere 6.5 for Machine Learning and Other HPC Workloads HOL-1947-01-EMT - Machine Learning Workloads in vSphere Using GPUs   Other articles Dell PowerEdge RAID Controller (PERC) S130 User’s Guide NVIDIA CUDA Installation Guide for Linux