Aiursoft Incident Investigation, Analysis, and Summary

This document records an NFS server failure and its recovery process caused by a defect in a ZFS backup script. The incident originated from the backup script failing to properly unmount the target dataset during snapshot transfer, which caused overlapping mount points between the source and target datasets. This led to data access anomalies and chaos in the vSphere data center, ultimately taking three hours to repair. The root cause was a misunderstanding of the `zfs send/receive` mechanism, which synchronizes mount point metadata. Additionally, the mount point displayed by `zfs list` is merely metadata and does not reflect the actual mount status. The lesson learned emphasizes that scripts in production environments must be thoroughly tested in a lab first, and a monitoring and alerting system based on Prometheus should be established. The improved new script adopts a "snapshot send - rollback - mount" strategy and enforces unmounting the target dataset before transfer. This ensures that data remains directly accessible via the file directory even after disk failures. After multiple validations, this solution effectively guarantees that at least two copies of the data are maintained, thoroughly resolving the mount conflict issue.