Backing Up your Data
It is always a good idea to have a back-up copy of data on your virtual machines. Instance snapshots are a great way to preserve a copy of the software configurations, configurations and profiles of a Virtual machine, but will not back-up the data stored in the secondary ephemeral storage drive or an attached volume. Volume Snapshots are a method of preserving the disk state of a volume in order to relaunch it in the future, but is also not an appropriate data back-up mechanism.
A data back-up strategy should include transferring data to another data storage location, such as your desktop or other research data storage.
The source for the back-up will be the secondary 'ephemeral' storage disk, or from volume block storage.
The destination for the back-up may be your local computer, or a data storage server at your research organisation.
Compressed File Transfers
The simplest back-up is to compress a directory and transfer it from your VM to your back-up destination.
The general comand structure to compress files:tar -cvpzf <NameOfArchive>.tar.gz <list of your files or folders>
To copy a directory ( /mnt/data ) into a single, compressed file ( data.tar.gz ):
tar -cvpzf data.tar.gz /mnt/data
The compressed file can be transferred to another computer using FileZilla (for a local computer back-up), or SCP to back-up on a remote server (see the Cloud Basics article: Transferring Data)
scp <Path_To_Source_File> <Path_to_Destination>
e.g. scp ~/data.tar.gz username@remote.host.edu.au:data/directory
or, scp -i path/to/key ~/data.tar.gz username@remote.host.edu.au:data/directory
Backing up with RSYNC
This is a utility that creates incremental backups, such that the most recent synced state of a source directory is 'mirrored' in a directory on the destination storage device.
Syncing is a more efficient way of backing up data, as after the first sync, only modifications to the files will be transferred.
- RSYNC must be installed on the source computer (the VM) and the destination computer.
- RSYNC is pre-installed on MacOSX
- On Ubuntu, enter
sudo apt-get install rsync
- On Windows, installation is through the Cygwin package and usage is more complicated.
The general command for creating or syncing the back-up:
rsync -av <source directory> <destination directory>
or,rsync -av -e -i <path-to-private-key> <source directory> <destination directory>
To sync from the Terminal app. on your local computer, from the directory you are syncing into:
rsync -av ubuntu@NNN.NNN.NNN.NNN:/mnt/data/ dataCopy/
(this syncs data from the VM)rsync -av dataCopy/ ubuntu@NNN.NNN.NNN.NNN:/mnt/data/
(this will restore data to the VM)
To sync from the VM command line to a remote server:rsync -av /mnt/data/ username@remote.host.edu.au:data/directory/
and to restore:rsync -av username@remote.host.edu.au:data/directory/ /mnt/data/
Backing up Volumes to the Object Storage
There are methods to save a copy of the data on a volume to the Nectar Object Storage. These methods vary between the Availability Zones, so please lodge a support ticketĀ for more information that is relevant to your situation.