Introduction to Cloud Storage

Your Nectar Virtual Machine uses a combination of data storage options, whether for reading or creating your research data, storing a local git repository, running the operating system, etc. This page introduces the common data storage systems you will come across in Nectar.

Root Disk

This storage is used predominantly for the operating system and other software installed on the virtual machine. Depending on the flavor of your instance it is typically either 10GB or 30GB in size. The user /home directories also tend to be on this device. 

Data stored on the root disk are persistent until the termination or rebuild of the virtual machine. If you reboot your virtual machine, the data remains intact and when you take a snapshot of your virtual machine and use it to launch a new instance, the entire root disk is copied. The root disk is fixed in size and is used by the operating system. Caution: if the root disk is completely filled (e.g. with log files or user data), your operating system may respond unpredictably. The name of this device as seen from the virtual machine is usually '/dev/vda' and it is mounted as '/'.

Volume Storage

Volume storage is the virtual equivalent of a USB drive. A USB drive retains your data, whether it is plugged in or not. Manipulating the data on a USB drive requires that it is plugged into a computer and that it is mounted by the operating system. Your USB drive can be unplugged and plugged into another (newer, bigger, better) computer, but your USB drive can only ever be plugged in to one computer at a time. 

Equivalently a volume in your Nectar project can retain your data, whether it is attached to an instance or not. Manipulating the data on the volume requires that is attached to an instance, and that the file systems is mounted by the operating system. Your volume can be detached and attached to another (newer, bigger, better) instance, but your volume can only ever be attached to one instance at a time. 

Learn more about managing Volume storage, for your Nectar project including allocations, booting and volume snapshots.

Ephemeral Storage

Ephemeral storage is a storage device whose life cycle is closely related to that of an instance: it is created along with the instance, and it is terminated at the same time too. It does not exist independently of its instance. Some past Nectar public flavors included Ephemeral Storage and some specialty private flavors at various nodes included ephemeral storage.

Read more about Ephemeral Storage

Object Storage

The Nectar Cloud Object Storage is not a traditional file system, but rather a distributed storage system for static data such as virtual machine images, photo storage, email storage, backups and archives. Having no central "brain" or master point of control provides greater scalability, redundancy and durability. When you put a file in the Nectar Cloud Object Store, three copies of your data are distributed to different hardware for extra data safety and performance.

Think about that dataset comprised of 2GB files that you read in and analyse many times, but in general it doesn't change. Or the images you want to use on the cloud. Those are a couple examples of perfect data for Object Storage. Objects are written to multiple hardware devices in the data center to ensure integrity, and great performance!

In general, the object store is great for data you write once and read many times, but not suitable for applications like databases. It's the safest place to put your data on the Nectar Research Cloud as multiple redundant copies of your data are made, and it has great performance. You can access the object store from anywhere on the Internet, and data from Object Storage can be transferred to and from your virtual machine with a variety of http-capable tools.

Object Storage is completely decoupled from your virtual machine, so even if you reboot, delete or crash your virtual machine, your Object Storage files will remain safe (unless you remove them yourself). Object Storage persists independently of the life of an instance.

Swift

Swift is the component that provides object storage for OpenStack. With your credentials and via a URL you can request Swift to reserve & create storage (called containers or buckets). Files (known as objects when stored in Swift) can then be uploaded and accessed similarly by your running virtual machines.

The Nectar implementation of Swift is geodistributed across Nodes of the Nectar Cloud so that availability does not rely on any one data center or network infrastructure. Each collection of Swift nodes/hardware is known as a region, which may or may not include a Swift proxy server (the Internet facing and serving component of Swift). With some Swift clients/APIs users can explicitly choose which proxy to connect to, this might be useful e.g. for speeding up writes to object storage by choosing the nearest proxy. Due to Nectar's Swift having multiple regions (some of which are Node private) some clients/APIs require explicit configuration of a default region, which should be "Melbourne" for most users.

This series of articles will help you get started with object storage in Nectar.