• Most of the data and the underlying disks still exist in the cluster before the restore and can be directly reused then.
  • There is a backupstore holding all volume data.
  • The setting is false. (It’s false by default.) Otherwise, users need to manually check if the data among volume replicas are consistent, or directly restore volumes from backup.
  • All settings and node & disk configs will be restored.
  • As long as the valid data still exists, the volumes can be recovered without using a backup. In other words, we will try to avoid restoring backups, which may help reduce Recovery Time Objective (RTO) as well as save bandwidth.
  • Detect the invalid or out-of-sync replicas as long as the related volume still contains a valid replica after the restore.

  • Restart all Kubernetes components for all nodes. See the above link for more details.

  • Kill all longhorn manager pods then Kubernetes will automatically restart them. Wait for conflicts in longhorn manager pods to disappear.

  • All volumes may be reattached. If a Longhorn volume is used by a single pod, users need to shut down then recreate it. For Deployments or Statefulsets, Longhorn will automatically kill then restart the related pods.

    • A volume is unchanged: Users don’t need to do anything.

    • The data is updated: Users don’t need to do anything typically. Longhorn will automatically fail the replicas that don’t contain the latest data.

    • A new volume is created: This volume will disappear after the restore. Users need to recreate a new volume, launch a single replica volume based on the replica of the disappeared volume, then transfer the data to the new volume.

    • For DR volumes: Users don’t need to do anything. Longhorn will redo a full restore.

    • Some operations are applied for a volume:

      • Snapshot: The snapshot info of the volume should be resynced once the volume is attached.
      • Replica rebuilding & replica removal:
        • If there are new replicas rebuilt, those replicas will disappear from the Longhorn system after the restoring. Users need to clean up the replica data manually, or use the data directories of these replicas to export a single replica volume then do data recovery if necessary.
        • If there are some failed/removed replicas and there is at least one replica keeping healthy, those failed/removed replicas will be back after the restoration. Then Longhorn can detect these restored replicas do not contain any data, and copy the latest data from the healthy replica to these replicas.
        • If all replicas are replaced by new replicas after the snapshot, the volume will contain invalid replicas only after the restore. Then users need to export for the data recovery.
      • Engine image upgrade: Users need to redo the upgrade.
      • Expansion: The spec size of the volume will be smaller than the current size. This is like someone requesting volume shrinking but actually Longhorn will refuse to handle it internally. To recover the volume, users need to scale down the workloads and re-do the expansion.
    • Notice: If users don’t know how to recover a problematic volume, the simplest way is always restoring a new volume from backup.

  • If the Longhorn system is upgraded after the snapshot, the new settings and the modifications on the node config will disappear. Users need to re-do the upgrade, then re-modify the settings and node configurations.