Recovery Options When You Cannot Access Your Instance

The best recovery option is to always keep a recent snapshot of your instance and/or data backups that you can revert to when there is problem with accessing an instance.

These access problems are typically caused by a user making a configuration change to the instance operating system that means the system will not boot correctly or that users are not able to login.

Such a problem might occur if a user has misconfigured the SSH service on the instance or if there is a misconfigured filesystem mount point. Re-launching an instance from a recent snapshot and restoring any other data from your backups is the most reliable approach. If you do not have a suitable snapshot or backups, there are some other options aimed at more advanced users that may help.

Operating System Recovery Mode

Nectar Ubuntu and CentOS images can be booted into recovery or emergency mode. These modes start a root shell with basic operating system services. It may then be possible to edit the misconfigured service, or perform other activities such as scheduling a file system check. There are two ways to boot into recovery mode, via the Dashboard action menu, and the VNC console.

Using the Nectar Dashboard action menu

On the main menu, go to `Project` > `Compute` > `Instances`.

For the instance which you need to rescue, click on the action menu, then click the `Rescue Instance` option. 

In the dialog box that appears, you can select an image. Best to select what you originally had on your instance, so you are familiar with it. Then click `confirm`. 

You will then see on the dashboard, your instance status in rescue mode. You should be able to now connect to it via SSH using the same details as previously. Note: If you selected a different image than on your original instance, your username has changed (e.g. fedora images have a fedora username). 

Once you have logged in via SSH, you can use commands to access your files and enact commands where necessary. 

In basic terms, in rescue mode, it's like the main hard drive of your computer being taken out, and plugged into another computer for access. To see what's currently available in your instance, while it's in rescue mode, input the lsblk command:

[fedora@playpen ~]$ lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda    252:0    0   4G  0 disk
├─vda1 252:1    0   1G  0 part /boot
└─vda2 252:2    0   3G  0 part /
vdb    252:16   0  30G  0 disk
└─vdb1 252:17   0  30G  0 part

sudo mount /dev/vdb1 /mnt

ls /mnt

Using the VNC Console on the Nectar Dashboard

If you haven't used the console previously, follow this tutorial first to set a password.

To access the Console, open the Nectar Dashboard and navigate to Compute -> Instances -> Select the relevant instance -> Console. When you get to the Instance Console tab, click the link Click here to show only console.

Click the Send CtrlAltDel button on the top-right of the page to reboot the instance without disconnecting the console. For CentOS images, be ready to press the Down Arrow key as the instance restarts. For Ubuntu be ready with the Shift key.

CentOS Emergency Mode

To enter CentOS emergency mode, be ready to press the Down Arrow key as the boot menu displays. Make sure the default boot item is selected (usually the first entry), then press the 'e' key to edit the selected item.

Cursor down to the line starting with linux16 and delete any console=ttyS... parameters, as highlighted. Be sure to leave the console=tty0 parameter as is.

At the end of the same line add the parameters rd.break enforcing=0.

Press Ctrl-x to start the system in emergency mode.

Your instance root filesystem will be mounted read-only under /sysroot. You can remount the filesytem in read-write mode as follows.

switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot

You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command.

sh-4.2# vi /etc/fstab

When you have completed the required changes/repairs, enable an SELinux relabel on reboot by touching the /.autorelabel file. This will ensure the SELinux contexts are set for any file changes made in emergency mode. Use the exit command to close the chroot environment, then again to reboot the instance.

sh-4.2# touch /.autorelabel
sh-4.2# exit
switch_root:/# exit

Ubuntu Recovery Mode

To enter Ubuntu recovery mode, hold the Shift key as the instance is reboots. If the GRUB boot menu does not appear, try again making sure you start holding the Shift key immediately after you click the Send CtrlAltDel button. If this doesn't work, press the ESC key multiple times right after clicking the CtrlAltDel button. When the GRUB boot menu appears, select Advanced options for Ubuntu and press Enter.

Then select Ubuntu, with Linux ... (recovery mode).

The Recovery Menu provides various recovery tools/options. To access the root shell prompt, you can select the root option and press Enter.

The root shell prompt has your instance root filesystem mounted in read-only mode.

You can remount the filesytem in read-write mode as follows.

root:~# mount -o remount,rw /

You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command.

root:~# vi /etc/fstab

When you have completed the required changes/repairs, use the exit command to close the shell and return to the Recovery Menu.

root:~# exit

In the Recovery Menu, select the resume option and press Enter to resume normal instance reboot. Press Enter again at the graphics drivers note.

OpenStack Rescue Mode

The OpenStack server rescue command can also be used to access your image in some cases. It boots from a fresh image of the image you originally used when you launched the instance, with a disk device configured for your image. You can use the disk device to mount your image and make the required configuration fixes. You will need to have the OpenStack server client installed locally.

Install the OpenStack command line client as described in the Manage instance via API Nectar Solutions article.

To put the server in rescue mode, first stop the instance.

openstack server stop <your instance id>

Check that the server has stopped using the server status command.

openstack server show <your instance id>

Once stopped, start the instance in rescue mode.

openstack server rescue <your instance id>

Check that the server is in rescue mode using the server status command. The server will be ready when vm_state is "rescued".

openstack server show <your instance id>

You should now be able to login with your Nectar SSH key file as usual.

ssh -i <your key file> <ec2-user|ubuntu>@<your instance ip address>
...
$

Use the fdisk command to get the disk device for your instance.

$ sudo fdisk -l
Disk /dev/vda: 2 GiB, 2147483648 bytes, 4194304 sectors
...
Device     Boot Start     End Sectors Size Id Type
/dev/vda1  *     2048 4192255 4190208   2G 83 Linux

Disk /dev/vdb: 30 GiB, 32212254720 bytes, 62914560 sectors
...
Device     Boot Start      End  Sectors Size Id Type
/dev/vdb1  *     2048 62914526 62912479  30G 83 Linux

In this example the required device is /dev/vdb1. You can mount the instance filesystem as follows.

$ sudo mount /dev/vdb1 /mnt

You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command. Note that you need to prefix filenames with /mnt to access the instance filesystem.

$ sudo vi /mnt/etc/fstab

When done, logout and restart the instance in normal mode.

openstack server unrescue <your instance id>