Disaster Recovery

This is a guide to setting up DR for Cortex Fabric.

This disaster recovery solution uses a third party Velero client.

Install Velero client

Install Velero client from this location.

Install the Velero Server in the same EKS cluster that is running Cortex using either their Helm charts or CLI. Bundling Velero Helm along with Cortex is a more reliable solution.

EKS Info

EKS-based customers must use the Velero plugin for AWS. Install the plugin at this step.

Set up the backup schedule

Set up backups of Cortex data at recurring intervals.

You may set up the backup schedule as a cron job by running the following command:

velero schedule create <SCHEDULE NAME> --schedule "0 7 * * *"

Performing recovery

When disaster happens, before starting the restoration process, you must change the setting to read-only at the backup destination to avoid data integrity loss.

1. Do this by running the following command:

   kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
       --namespace velero \
       --type merge \
       --patch '{"spec":{"accessMode":"ReadOnly"}}'

2. Restore with a backup by running:

   velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

3. When the restoration is complete, change the storage backup location to read-write mode by running:

   kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
      --namespace velero \
      --type merge \
      --patch '{"spec":{"accessMode":"ReadWrite"}}'