Aiursoft Nextcloud Incident Investigation, Analysis, and Summary
On October 4, 2022, Aiursoft Nextcloud experienced a severe incident that resulted in a two-hour service interruption and partial data loss. The root cause was the accidental loss of the NFS backup disk mount following the storage and compute separation migration. This caused the backup task to write large amounts of data to the system disk, instantly filling up the 64GB system disk, which led to system process paralysis and failed restarts. During the investigation, by mounting the hard drive to another server for analysis, it was discovered that system disk space was being abnormally occupied. It was ultimately confirmed that the backup script had erroneously written data to the local path when the mount failed. This incident revealed potential risks in the backup mechanism when configuration errors occur, as well as the lack of monitoring for mount point status. Improvement measures include enforcing mount status verification before backups, establishing real-time alert mechanisms for disk remaining space and mount status, and optimizing the backup script to prevent accidental writes to the local disk, thereby enhancing system stability and data security.