In high-performance computing environments that support core business operations, data backup is not merely physical redundancy; it is also a fault-tolerance mechanism against logical errors. This article documents a versatile disaster recovery solution implemented on an AnduinOS server, balancing IO efficiency with the ability to trace back through historical versions.

Many people use rsync for backups. rsync performs incremental transfers on each run, minimizing bandwidth and resource usage.

However, if rsync runs too infrequently, new data may not be backed up to the backup disk in a timely manner. If rsync runs too frequently, accidentally deleted files may also be backed up.

Many people use snapshot-based backups, such as ZFS snapshot send. The result is: even if I haven't modified a single file, transferring the snapshot wastes a massive amount of resources. This forces you to back up very lazily.

Moreover, snapshot-based backups require the data disk to also be ZFS. I have always felt that ZFS performance is not great and that it consumes a lot of memory. I don't like using it. The data disk could be any type, even NTFS.

So, is there a versatile solution: one that incurs a small cost per backup while still allowing you to trace the history of file changes?

This is the heterogeneous part introduced in this article:

1. Background: Analysis of the Limitations of Traditional Disaster Recovery Paradigms

1.1 Performance-First: The Selection Logic for the Main System Filesystem

To ensure the ultimate IO response in the production environment, the primary storage system avoids file systems that employ complex "Copy-on-Write" (COW) mechanisms. Although ZFS is feature-rich, its metadata overhead under high-frequency random write scenarios can become a bottleneck.

Therefore, the primary storage utilizes the Ext4 file system, combined with bcache and NVMe caching. This architecture achieves optimal throughput, but it does not natively support efficient historical version management.

1.2 Assessment of the Deficiencies of Traditional Backup Methods

  1. Physical Image Backup (dd): For datasets on the scale of tens of trillions of bytes (10TB+), a full physical copy generates intense IO load.

    • Hardware Wear: Frequent full-disk scans shorten the lifespan of mechanical hard drives.
    • Time Cost: The backup window is too long to meet business continuity requirements.

  2. Logical Limitations of Incremental Sync (rsync): Simple incremental mirroring (rsync --delete) cannot defend against logical errors.

    • Sync Conflicts: If an accidental deletion occurs on the production side, this operation will rapidly propagate to the backup side.
    • Frequency Trade-off: Reducing sync frequency (e.g., once a day) increases the data loss window; increasing frequency (e.g., every 5 minutes) causes logical errors to destroy the backup copy in near real-time.

2. Solution Design: Decoupled Architecture for Heterogeneous File Systems

To address the aforementioned challenges, we achieved a balance between performance and security through architectural decoupling: primary storage prioritizes efficiency, while backup storage provides fault tolerance.

2.1 Core Logic Derivation

  • Execute Ext4 on the production side: Enjoy native high-performance capabilities.
  • Execute ZFS on the backup side: Leverage its excellent snapshot features to implement versioned management.

2.2 Multi-Dimensional Disaster Recovery Features

  1. High-frequency differential synchronization: Perform incremental synchronization every 30 minutes using rsync to prevent data loss caused by physical failures.
  2. Instantaneous state locking: Trigger a ZFS snapshot immediately after synchronization completes.
    • Space efficiency: ZFS snapshots only record data block differences and do not consume additional physical space.
    • Logical defense: Snapshots act as read-only nodes, providing "time travel" capabilities to recover from logical errors such as accidental deletions.

3. Production Environment Deployment Practices (proart)

3.1 Building a ZFS Storage Pool

Build a storage pool with LZ4 compression capabilities on the backup medium (/dev/sdd) to optimize storage density.

sudo apt update && sudo apt install -y zfsutils-linux
sudo zpool create -f backup_pool /dev/sdd
sudo zfs set compression=lz4 backup_pool
sudo zfs set atime=off backup_pool
sudo zfs create -o mountpoint=/mnt/backup backup_pool/swarm_backup

3.2 Configuring Automated Snapshot Policies (Sanoid)

Implement a tiered snapshot retention policy using sanoid to ensure coverage of time windows at different granularities.

Configuration file: /etc/sanoid/sanoid.conf

[backup_pool/swarm_backup]
    use_template = production
    recursive = yes

[template_production]
    hourly = 24    # 保留 24 小时内的小时级快照
    daily = 30     # 保留 30 天内的日级快照
    weekly = 8     # 保留 8 周内的周级快照
    monthly = 12   # 保留 12 个月的月级快照
    autosnap = yes
    autoprune = yes

4. Operations and Emergency Response Manual

4.1 Operational Status Monitoring

  1. Initial Synchronization Monitoring: It is recommended to run large-scale data migrations within a tmux session, monitored via tmux attach -t initial-backup.
  2. Concurrency Conflict Control: The script includes a built-in file locking (flock) mechanism. Use tail -f /tmp/backup_last_run.log to check if tasks were skipped due to overlap.
  3. Snapshot List Retrieval: 
    zfs list -t snapshot -o name,creation -S creation backup_pool/swarm_backup

4.2 Data Extraction and Disaster Recovery

1. Fine-Grained File Extraction (Non-Destructive): ZFS's hidden snapshot directory (.zfs/snapshot) allows operators to access historical versions with zero overhead.

# 访问特定时间节点的镜像目录
ls /mnt/backup/.zfs/snapshot/auto-20260308-1500/

# 直接从备份副本中恢复指定文件
cp /mnt/backup/.zfs/snapshot/auto-20260308-1500/path/to/file /swarm-vol/

2. Volume-Level Full Rollback: In the event of catastrophic logical corruption, the entire dataset can be rolled back with a single click.

# 警告：此操作将抹除快照点之后的所有变更
zfs rollback backup_pool/swarm_backup@auto-20260308-1500

4.3 Storage Mount Persistence Description

Do not configure ZFS entries in /etc/fstab. ZFS has its own independent mount management mechanism and automatically detects and mounts filesystems during boot via zfs-import-cache.service. Forcing configurations in fstab may lead to mount conflicts or boot failures.

5. Architecture Evaluation and Cost Analysis

  1. Computational Resources: Traversing a large-scale file tree with rsync consumes a certain number of CPU cycles, but thanks to cache warming, the impact on business operations is minimal.
  2. Storage Overhead: Frequent snapshots may lead to a certain degree of disk fragmentation. However, in backup storage scenarios, the benefits of versioning far outweigh the performance degradation caused by fragmentation.
  3. Risk Coverage: This solution provides comprehensive coverage against both physical damage and logical misoperations.

Conclusion: By implementing architectural decoupling between heterogeneous file systems, we not only maintained high performance on the production side but also built a reliable "time machine" mechanism on the backup side. For business systems that host core assets, this is a highly valuable engineering reference for achieving balance.