How should I describe Nectar computing hardware in research publications?

Nectar users occasionally ask us how to describe the compute hardware of a Nectar instance.  For example, what is the CPU model or clock speed of an instance.  The typical motivation for these questions is to support performance measurements for a novel application or algorithm in a research publication.

Unfortunately, it is almost impossible to give an answer that will stand up to scrutiny.

Problem 1

The compute hardware on which an instance is running is highly variable within and between Availability Zones.  Across Nectar Research cloud, we are running both AMD and Intel processors, with various models and clock speeds, ranging from almost new to up to six years old.  On top of that, RAM speeds may be different, local disks (or SSDs) and disk controllers performance may be different, network performance maybe different.  Then there are significant differences in the way that volume storage, object storage and NFS file storage are implemented, leading to significantly different levels of file performance.

There is no up to date Nectar-wide source of information on hardware and service level performance, and no easy way for you (or us) to find out the performance of a given compute host. It is possible to find out some data about the hardware that an instance is currently running; for example using "cat /proc/cpuinfo".  However there are caveats with this approach:

• The information is provided by the virtualization layer, and *might not be entirely accurate*.
• This does not take account of possible "over-commit" of CPUs and Physical RAM.

Problem 2

Your instance is a virtual machine.  While we use a flavor of virtualization that means that your application runs using hardware instructions, virtualization means that there is a performance hit on network and disk I/O, and also on swapping / paging.  Plus a small percentage of the CPU and memory will be used for running the host operating system, the hypervisor and management services.

Problem 3

VCPU overcommit.  You will have noticed that Nectar flavors are described in terms of VCPUs rather than cores or processors.  The V stands for "virtual" as in virtual CPUs.  

It is standard practice for cloud computing providers to optimize the utilization (and economics) of the compute hardware by over-committing the hardware.  For example, if the CPUs on a compute node are overcommitted by a factor of 2, 1 VCPU will give you about 1/2 of a physical core if all of the resources are being used at 100% utilization. However, the hardware is typically not that busy.  If half of the VCPUs are idle, the other half will perform at close to 100% of a physical core.

In the Nectar cloud, the amount of overcommit of CPUs varies depending on the flavor of the instance, and policy decisions made by the local Nectar Node operators.  These policies are not published, but the overcommit ratio for VCPUs to CPUs can be as high as 4:1.  

Normally this isn't a problem.  Most instances are idle most of the time.  However, if an instance with an over-committed flavor is running on a compute node that is busy, its compute performance could suffer. Unfortunately, it is next to impossible for an instance to monitor the effects of overcommit.

Problem 4

The performance of a Nectar instance may be influenced by the behavior other instances on the same compute node. It can also be influenced by the load on external services; e.g. the load on the shared file servers that hold instance's file system.  (We refer to this as the "noisy neighbour" effect.)

In Summary

These issues make it next to impossible to compare the observed performance of Nectar instances to other systems.  Even comparing different Nectar instances has problems.