Install your own NFS ESXi storage server

Background

Recently, I was building a VMware ESXi cluster, also known as vSphere.

My rack structure is as follows. I am using my own server as shared NFS storage.

This post is about how to configure this server.

file

Why am I doing this?

We all know that Vsphere clusters require separate storage so that ESXI hosts can migrate virtual machines between multiple hosts.

There are professional solutions like Dell PowerStore. However, those machines are really really expensive, about 20,000 USD.

So how can we get a similar solution as an ESXI storage server with only around 800 USD?

Step 1 - Buy hardware

1.1 Buy a server (400USD)

My personal suggested hardware is:

SKU	CPU	RAM	SATA SSD	NVME SSD	HDD
Dell R730XD 12 disks	Intel E5-2620v3 * 2, 2.4GHz, 12 Cores, 24 Threads.	16GB x 2 DDR4	256GB x 1	2TB	2TBx12

<div style="display:inline-block;white-space:pre;background-color:#4C001E;font-family:'Cascadia Code',monospace;font-size:12pt;padding:4px;">anduin@NFS:~$ neofetch ,met$$$$$gg. anduin@NFS ,g$$$$$$$$$$$$$$$P. ---------- ,g$$P" """Y$$.". OS: Debian GNU/Linux 11 (bullseye) x86_64 ,$P'              `$$. Host: PowerEdge R730XD ',$$P ,ggs. $$b:   Kernel: 5.10.0-18-amd64 d$$' ,$P"'   .    $$$ Uptime: 1 day, 4 hours, 34 mins $$P d$' , $P    Packages: 571 (dpkg)  $: $.   -    ,d$' Shell: bash 5.1.4 $;      Y$b._   _,d$P'      Resolution: 1024x768  Y$. ."Y$$$$P"' Terminal: /dev/pts/0 $$b      "-.__              CPU: Intel Xeon E5-2620 v3 (24) @ 3.200GHz   Y$$ GPU: 0b:00.0 Matrox Electronics Systems Ltd. G200eR2 Y$$.                      Memory: 650MiB / 31816MiB      $$b. Y$$b.           "Y$b. `"""</div>

The server costs about 400USD.

1.2 Buy a SATA SSD (30 USD)

You need to purchase a SATA SSD to install the operating system. This is because the Dell R730XD only has 3.5-inch HDD slots. I really dislike installing my OS on an HDD.

Additionally, the Dell R730XD is a very old server that does not recognize NVMe SSDs before booting. Therefore, you cannot install your OS on an NVMe SSD.

That's why it is frustrating that you have to buy another SATA SSD solely for the OS installation.

file

1.2 Buy NVME (300USD, Optional)

You will need some NVMe.

The NVMe disk is for machines that require high random access speeds, as RAID does not speed up random access.

I am using the SAMSUNG 980 PRO as NVMe storage. It costs about 300 USD (2100 CNY).

file

1.3 Buy some disks (400USD)

You will need some HDDs. For example, I'm using the TOSHIBA P300.

12 disks cost about 400 USD.

file

1.4 Buy NVME to PCIE converter (3USD)

Yes. You need an NVMe to PCIe adapter to connect your NVMe SSD to your server!

Usually, this is very inexpensive. About 3 USD.

file

1.5 Buy SATA to PCIe converter (3 USD)

Yes. You also need to buy a SATA to PCIE converter, so you can connect your SATA SSD to PCIE port.

file

1.6 Buy fiber network adapter (80USD)

You need to buy some fiber network adapter. It is suggested to buy at least 3.

I am using Intel X520 DA2. Make sure it supports 10Gbps.

file

1.7 Assemble your server like this!

file

Step 2 - Configure iDRAC

Step 2.1 Upgrade your firmware

First, you need to upgrade the firmware.

You need to use a cable to connect the iDRAC port to your router. Try to find its IP address and sign in.

Connect this port to your LAN. Try to get the IP address.

file

Default username: root. Default password: calvin.

To upgrade the firmware, you need to download it from Dell.

Download R730 latest Firmware

Download the latest BIOS and iDRAC. Download the Windows x64 installer, and unzip it.

file

Extract it with 7zip.

file

And upload the firming.d7 file to the iDRAC portal.

file

Step 2.2 Fix fan issue

Enable IPMI over LAN

Make sure this is enabled:

file

Download IPMI tools

For Windows, download it here.

After downloading, install it and add the extracted path to system environment variable PATH.

For Linux, try:

sudo apt install ipmitool

You can try to run it after installing:

file

Fix your server

Here,

-H follows the host name. 192.168.50.20 as an example.
-U follows the user name. root as an example.
-P follows the password. calvin as an example.

Change these values to your own. And run:

ipmitool  -I lanplus -H 192.168.50.20 -U root -P calvin raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

This will disable the third-party PCIE default cooling response.

After that, your server is fixed. The fans have returned to normal operation.

file

Why that command?

# Query `Dell's Third-Party PCIe card based default system fan response` status
ipmitool raw 0x30 0xce 0x01 0x16 0x05 0x00 0x00 0x00

response like below means Disabled

16 05 00 00 00 05 00 01 00 00

response like below means Enabled

16 05 00 00 00 05 00 00 00 00

To disable it:

0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x00

To disable it: (Jets off)

0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

Manual fan control?

Yes, you can perform manual fan control.

To enable this:

192.168.18.7 raw 0x30 0x30 0x01 0x00

The speed is a hexadecimal number, from 00 to 64. You may need a hex converter.

https://www.rapidtables.com/convert/number/hex-to-decimal.html

64 is 100%
50 is 80%
00 is 0%.

For example, to set fan speed to 80%,

0x30 0x30 0x02 0xff 0x50

Step 3 - Configure RAID 6

After setting up your fan and iDRAC, you can set up your RAID now.

It is recommended to configure a RAID 6 array because it is safe.

You need to open the terminal of your server. Reboot it, and press F2 to enter RAID settings.

file

You need to create a new virtual disk. Select the relevant physical disks and RAID mode, and then initialize the disk.

file

Step 4 - Install Debian

Step 4.1 Download and burn Debian installer

Now download the Debian OS ISO file.

You can download it here.

file

I'm using Rufus on Windows to burn that ISO to a USB stick.

Make sure you select DD mode when burning the USB stick!

Step 4.2 Install the Debian OS.

Now you can install the Debian OS.

Simply insert the USB stick into your server and boot from it.

file

Follow the instructions to complete the Debian OS installation.

file

Please note this: Do NOT install your OS to your NVME!!! It won't boot!

Do install the OS to the SATA SSD!

file

After installing the OS, you need to configure authentication for best practices.

You can follow the instructions here if you are not familiar with Linux.

What need to do after you get a new Linux?

Step 5 - Mount disks

Run command sudo fdisk -l to locate the disk:

sudo fdisk -l

file

Now we can create a new partition. I'm using sda as an example in it.

sudo fdisk /dev/sda

Tips:

n – Create partition
p – print partition table
g - reset as GPT partition table
d – delete a partition
q – exit without saving the changes
w – write the changes and exit.

After creating, we can format it.

mkfs.ext4 /dev/sda1

After formatting, you need to get its UUID.

sudo blkid | grep UUID=

The output will look like this:

file

Copy the UUID of the PARTITION that appears after UUID, not the PARTUUID!

For example: 3238a745-7026-47c2-82a3-e48a4d31556b.

Now we need to mount it. For example, to mount the disk 3238a745-7026-47c2-82a3-e48a4d31556b to /mnt/random_store.

First, edit the /etc/fstab file:

sudo vim /etc/fstab

Add this line:

UUID=3238a745-7026-47c2-82a3-e48a4d31556b /mnt/random_store     ext4    defaults,noatime,nofail 0       0

So it will be like:

file

Try to mount it:

sudo mkdir /mnt/random_store
sudo mount /mnt/random_store

You can verify the mappings:

sudo lsblk

file

Step 6 - Enable NFS Server

Step 6.1 Install NFS Server

sudo apt update
sudo apt install nfs-kernel-server -y

Step 6.2 Expose an NFS path.

For example, we want to expose /mnt/nfs_share as an NFS path.

sudo mkdir -p /mnt/nfs_share
sudo chown -R nobody:nogroup /mnt/nfs_share/
sudo chmod 777 /mnt/nfs_share/

And edit the configuration:

sudo vim /etc/exports

Add this:

/mnt/nfs_share 172.16.1.0/24(rw,sync,no_subtree_check)

The 172.16.1.0/24 is the subnet that allows connection to this server. This is important; set it as the network for your ESXi to mount it!!

file

Finally:

sudo exportfs -a
sudo systemctl restart nfs-kernel-server

If you have ufw enabled, don't forget to expose port 2049.

Step 7 - Configure network

Step 7.1 - Connect your NFS server to your ESXi server

file

Establish the connection between your NFS server and your ESXI.

As shown, the Storage network in the picture.

file

Why do I need to install an additional DHCP server?

Because I do not want my VM's traffic to mix with storage traffic. While this is possible, it will reduce performance.

I use fiber to connect the ESXI host directly to the storage server. This provides the best performance.

However, I have multiple ESXI hosts. Therefore, I need to connect each one.

For ease of management, I can bridge multiple fiber optic ports into a subnet and enable the DHCP server within that subnet.

Step 7.1 Bridge all your fiber ports

sudo apt install bridge-utils

To view all connected network adapters:

ifconfig

Now bridge all fiber ports.

ip link add name bridge0 type bridge
ip link set dev bridge0 up
ip link set dev enp130s0f0 master bridge0
ip link set dev enp130s0f1 master bridge0
ip link set dev enp5s0f0 master bridge0
ip link set dev enp5s0f1 master bridge0

Don't forget to assign an IP to your bridge:

sudo ip addr add 172.16.1.1/24 brd + dev bridge0

You can enable Jumbo frames here:

sudo ip link set enp5s0f0 mtu 9000
sudo ip link set enp5s0f1 mtu 9000
sudo ip link set enp130s0f0 mtu 9000
 sudo ip link set enp130s0f1 mtu 9000 # All interfaces...
sudo ip link set bridge0 mtu 9000
sudo ip link show | grep mtu

Step 7.2 Start DHCP server

First, install your DHCP server.

sudo apt install isc-dhcp-server

Now edit the configuration:

sudo vim /etc/dhcp/dhcpd.conf

My configuration is:

option domain-name "storage.network";
option domain-name-servers ns1.example.org, ns2.example.org;

default-lease-time 600;
max-lease-time 7200;

subnet 172.16.1.0 netmask 255.255.255.0 {
 range 172.16.1.100 172.16.1.200;                   # My ESXIs.
 option routers 172.16.1.0;                              # A fake router.
 option domain-name-servers 172.16.1.0;      # A fake DNS
 option domain-name "storage.network";
}

file

Finally, you need to make sure your DHCP server doesn't disrupt your management network.

Make the DHCP server only listen to the bridge:

sudo vim /etc/default/isc-dhcp-server

Edit the content according to your bridge:

INTERFACESv4="bridge0"
INTERFACESv6=""

file

Don't forget to restart the DHCP service.

sudo systemctl restart isc-dhcp-server.service

Step 8 - Install cockpit for monitoring

Cockpit is a web portal that helps you monitor your server status. Installing it is simple.

file

Step 8.1 - Install cockpit

Run:

sudo apt install cockpit -y

Step 8.2 - Configure Authentication

Ensure that a user other than root is in the sudo group.

Please follow the instructions from: Best practice for authentication

Step 8.3 - Install navigator plugin

sudo apt install -y jq
latestUrl=$(curl https://api.github.com/repos/45Drives/cockpit-navigator/releases/latest | jq -r '(.assets[] | select(.browser_download_url | contains(".deb"))).browser_download_url')
echo "Latest download url is $latestUrl"
wget -O /home/anduin/install.deb $latestUrl
sudo dpkg -i /home/anduin/install.deb
rm /home/anduin/install.deb

Step 8.4 - Try cockpit on your browser!

Open the browser and navigate to: https://host:9090. Ignore certificate issues.

file

Don't forget, we are using cockpit to monitor our network and disk usage.

file

Step 9 - Mount the server to Vsphere

Step 9.1 Prepare your host to mount NFS storage

First, log in to your ESXi host.

Add a new virtual switch. Select the fiber interface as the uplink.

file

Name it, select the link, and save.

file

If you want to enable Jumbo frames, set the MTU to 9000:

file

Then add a new port group. Select the related storage virtual switch Storage switch.

file

Finally, add a new VMkernel NIC. Select the Storage network.

file

Step 9.2 Mount the NFS storage

Go to your VCenter server. Select adding a new store.

file

Select the type.

file

Select the version.

file

Enter the server details:

file

And finally, it works!

file

Step 10 - Benchmark your performance!

Of course CrystalDiskBench!

file

Don't forget to keep an eye on your network usage and disk usage!

10Gbps network should have max speed around 1200MB/s.

file

Background

Why am I doing this?

Step 1 - Buy hardware

1.1 Buy a server (400USD)

1.2 Buy a SATA SSD (30 USD)

1.2 Buy NVME (300USD, Optional)

1.3 Buy some disks (400USD)

1.4 Buy NVME to PCIE converter (3USD)

1.5 Buy SATA to PCIe converter (3 USD)

1.6 Buy fiber network adapter (80USD)

1.7 Assemble your server like this!

Step 2 - Configure iDRAC

Step 2.1 Upgrade your firmware

Step 2.2 Fix fan issue

Enable IPMI over LAN

Download IPMI tools

Fix your server

Why that command?

Manual fan control?

Step 3 - Configure RAID 6

Step 4 - Install Debian

Step 4.1 Download and burn Debian installer

Step 4.2 Install the Debian OS.

Step 5 - Mount disks

Step 6 - Enable NFS Server

Step 6.1 Install NFS Server

Step 6.2 Expose an NFS path.

Step 7 - Configure network

Step 7.1 - Connect your NFS server to your ESXi server

Step 7.1 Bridge all your fiber ports

Step 7.2 Start DHCP server

Step 8 - Install cockpit for monitoring

Step 8.1 - Install cockpit

Step 8.2 - Configure Authentication

Step 8.3 - Install navigator plugin

Step 8.4 - Try cockpit on your browser!

Step 9 - Mount the server to Vsphere

Step 9.1 Prepare your host to mount NFS storage

Step 9.2 Mount the NFS storage

Step 10 - Benchmark your performance!

Comments (1)