On the morning of October 4, 2022, between 10:00 and 12:00, a major incident occurred with Aiursoft Nextcloud. Our service was interrupted for two hours, and some data was lost. Availability was also severely degraded on that day.

incident

It took us approximately two hours to restore these services and data. Below is the detailed process.

Background

Aiursoft Nextcloud is a NAS storage service operated by Aiursoft. It operates on an invitation-only membership basis. Invited members can receive 200GB of storage space and enjoy features such as To do, Deck, Talk, Notes, and Passwords.

Aiursoft Nextcloud runs on the Ubuntu system. The system is a VMware virtual machine running on a VMware ESXi host.

Aiursoft Nextcloud itself is served by Apache. An additional layer of Caddy is used for reverse proxying. Caddy runs on a separate virtual machine.

To expose services to the public internet, Aiursoft Nextcloud also utilizes the FRP service.

This host recently completed a migration to separate storage from computation.

Aiursoft Nextcloud performs automatic backups daily at 2:00 AM (UTC). The backup destination is another NFS server. User files are copied to the remote NFS using rsync.

Preliminary Accident Investigation

At 10:00 AM, I noticed that I couldn't access Nextcloud. The server was timing out. I also received an incident alert. Since I had just woken up, I immediately started troubleshooting.

First, I tried accessing it via the internal network, but it still didn't work. This confirmed that the issue wasn't related to the proxy or the network. Caddy was reporting 502 errors, while Apache was reporting 500 errors.

I planned to investigate the server via SSH, but found that the server was unresponsive. The SSH service was also down.

I then tried to manage it using Cockpit, but that service was also unresponsive. It felt as if the server was completely dead.

Given that both management options were completely unresponsive, I ultimately downgraded to using VNC for management. I used the VNC feature provided by vSphere.

However, VNC froze on the login page. My initial conclusion is that the server has completely hung. I'm even wondering if something absurd, like a cosmic ray hitting the memory, might have happened...

Restart

Considering that I had no way to deal with this virtual machine. It was completely unresolvable, so the only option left was to perform a hard power-off and restart. Before restarting, I took a memory snapshot.

After the hard power-off and restart, the server still failed to boot normally. During startup, a large number of services failed to load correctly, and the process was extremely slow.

This phenomenon left me completely baffled, especially since the point where it got stuck was different every time... It was highly confusing and frustrating.

Suspect Storage

The recent change we made was to implement separation of storage and compute. This led us to suspect that there might be some unusual changes in the connection between the physical servers and the storage devices. We suspected that there could be an issue with the fiber optic connection from ESXi to NFS.

However, it also seemed like other virtual machines were behaving abnormally, though we haven't yet identified a specific scope or gathered concrete evidence.

During previous testing, we discovered that if the connection from ESXi to NFS becomes unstable, the issue doesn't manifest immediately. ESXi has aggressive bidirectional caching.

Rollback

By this point, extensive research had been conducted without any definitive conclusion. There was no effective solution for dealing with this virtual machine.

At that time, there was an urgent need to restore nextcloud first, and essentially all conceivable methods had been attempted. Ultimately, out of necessity, the virtual machine backup was rolled back.

Strangely, everything functioned normally after the rollback. The newly created virtual machine also operated without any issues. It appeared unrelated to the recent changes involving storage and compute separation.

Mounting the Hard Drive on Another Machine

Since the system wouldn't boot at this point, I adopted a different approach: I removed the failed server hard drive, mounted it on another server, and then inspected it.

I found that the hard drive was full.

Let me clarify something here. This Nextcloud server has three drives in total: a 64GB system drive, a 1TB data drive, and a 1TB backup drive. Both the data drive and the system drive are remotely mounted NFS paths.

Both the data drive and the backup drive were normal, with about 30% usage. Only the system drive was at 64GB/64GB.

So the issue was that the hard drive was full. But why would the system drive fill up? None of the Nextcloud data was stored on the system drive. Yesterday, the system drive was only using 19GB, so why did it suddenly jump to 64GB today?

Why is the hard drive full?

Naturally, I ran this first:

du / -d 1 -h

The results after running were even more baffling. The results showed that / only occupied 19GB in total, not considering /mnt. This matched yesterday's usage data.

So, could the void have filled up the 64GB hard drive?

I ran du everywhere, checking each directory, and everything seemed normal. No abnormally large files appeared. Yet the hard drive was full.

I thought about what on the server could potentially write large amounts of files. The only possibility was the backup service.

Backup Service

Background: Nextcloud is backed up at 2:00 AM UTC. The backup method is as follows: copy the data disk to the backup disk.

  • The data disk is located at /mnt/data
  • The backup disk is located at /mnt/backup

One interesting issue is this: if /mnt/backup is not correctly mounted during the backup, then /mnt/backup becomes just a folder on the system disk. In this case, running the backup program would copy 300GB of data onto the system disk, causing it to fill up.

Upon closer inspection, this was indeed the cause.

After unmounting the backup disk, it was found that 40GB of data had filled up the path where the backup disk is located: /mnt/backup.

sudo umount /mnt/backup

Clearing it immediately and remounting restored 19 GB on the system disk.

Incident Scenario Reproduction

Normally, /mnt/backup is the backup drive. Because it is for backups, it is often neglected in terms of monitoring, so if it fails, no one notices.

Recently, we implemented a separation of storage and compute. The storage device was moved.

During the move, the connection may have experienced a momentary interruption, causing the mount of /mnt/backup to be lost.

At that point, /mnt/backup became a regular directory located on the system disk.

No one noticed this until this morning during the backup process. Nextcloud attempted to copy all data to /mnt/backup, which filled up the system disk.

Once the system disk was full, the impact on operations was catastrophic. All processes were unable to write files normally, and the system could not restart or shut down correctly.

In this state, restarting would actually result in the loss of data still in the memory cache, causing a major disaster. This was especially severe because a large amount of data was stuck in memory and had not yet been written to the disk.

Lessons Learned from This Incident

  1. Backups may appear safe, but they are actually highly dangerous. This is because backups involve a large amount of disk writes. A misconfiguration can lead to writing to the wrong disk.
  2. A monitoring and alerting system for remaining disk space must be established. The danger of a full disk is no less than that of disk corruption.
  3. On Linux, if a temporary connection issue occurs on a disk mounted under /mnt, the mount may be interrupted. In this case, /mnt/backup will become a regular folder under / and may be handled incorrectly.

Improvements from This Incident

  • Before running a scheduled backup, verify whether /mnt is actually a physical disk
  • Set up monitoring alerts to check that the disk is correctly mounted and that the remaining capacity is healthy. Trigger an alert when the remaining capacity falls below 10%.

New Backup Script Reference

set -e

if sudo df -Th | grep -q "/mnt/Backup"
then
        echo "Mounted!"
        # Backup
        sudo rm "/mnt/Backup/backups/$(ls /mnt/backup/backups/ | grep nextcloud-dirbkp | head -n 1)" -rvf
        sudo rsync -Aavx /var/www/html/nextcloud/ /mnt/backup/backups/nextcloud-dirbkp_`date +"%Y%m%d"`/
        sudo rsync -Aavx /mnt/datastorage /mnt/backup/dir/
        sudo mysqldump --single-transaction -h localhost -u root nextcloud > /home/anduin/temp.bak
        sudo mv /home/anduin/temp.bak /mnt/backup/database/nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
else
        echo "Not mounted!"
        # Alert.
        echo "Backup failed! `date +"%Y-%m-%d %H:%M:%S"`" | sudo tee -a /etc/motd
        exit;
fi