The best recovery option is to always keep a recent snapshot of your instance and/or data backups that you can revert to when there is problem with accessing an instance.
These access problems are typically caused by a user making a configuration change to the instance operating system that means the system will not boot correctly or that users are not able to login.
Such a problem might occur if a user has misconfigured the SSH service on the instance or if there is a misconfigured filesystem mount point. Re-launching an instance from a recent snapshot and restoring any other data from your backups is the most reliable approach. If you do not have a suitable snapshot or backups, there are some other options aimed at more advanced users that may help.
Operating System Recovery Mode
Nectar Ubuntu and CentOS images can be booted into recovery or emergency mode. These modes start a root shell with basic operating system services. It may then be possible to edit the misconfigured service, or perform other activities such as scheduling a file system check. There are two ways to boot into recovery mode, via the Dashboard action menu, and the VNC console.
Using the Nectar Dashboard action menu
On the main menu, go to `Project` > `Compute` > `Instances`.
For the instance which you need to rescue, click on the action menu, then click the `Rescue Instance` option.
In the dialog box that appears, you can select an image. Best to select what you originally had on your instance, so you are familiar with it. Then click `confirm`.
You will then see on the dashboard, your instance status in rescue mode. You should be able to now connect to it via SSH using the same details as previously. Note: If you selected a different image than on your original instance, your username has changed (e.g. fedora images have a fedora username).
Once you have logged in via SSH, you can use commands to access your files and enact commands where necessary.
In basic terms, in rescue mode, it's like the main hard drive of your computer being taken out, and plugged into another computer for access. To see what's currently available in your instance, while it's in rescue mode, input the lsblk command:
[fedora@playpen ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 252:0 0 4G 0 disk ├─vda1 252:1 0 1G 0 part /boot └─vda2 252:2 0 3G 0 part / vdb 252:16 0 30G 0 disk └─vdb1 252:17 0 30G 0 part
Above we can see, a small main boot drive (the vda1), and the root disk from our rescued instance (vdb1).
To access our root disk, we need to mount it. Here we are attaching it to the /mnt path.
sudo mount /dev/vdb1 /mnt
To see available files from your original root disk, type:
ls /mnt
You should be able to access and see your files now! Now your original root disk or hard drive is mounted, you should be able to see it in software like Filezilla as well.
Using the VNC Console on the Nectar Dashboard
If you haven't used the console previously, follow this tutorial first to set a password.
To access the Console, open the Nectar Dashboard and navigate to Compute -> Instances -> Select the relevant instance -> Console. When you get to the Instance Console tab, click the link Click here to show only console.
Click the Send CtrlAltDel button on the top-right of the page to reboot the instance without disconnecting the console. For CentOS images, be ready to press the Down Arrow key as the instance restarts. For Ubuntu be ready with the Shift key.
CentOS Emergency Mode
To enter CentOS emergency mode, be ready to press the Down Arrow key as the boot menu displays. Make sure the default boot item is selected (usually the first entry), then press the 'e' key to edit the selected item.
Cursor down to the line starting with linux16 and delete any console=ttyS... parameters, as highlighted. Be sure to leave the console=tty0 parameter as is.
At the end of the same line add the parameters rd.break enforcing=0.
Press Ctrl-x to start the system in emergency mode.
Your instance root filesystem will be mounted read-only under /sysroot. You can remount the filesytem in read-write mode as follows.
switch_root:/# mount -o remount,rw /sysroot switch_root:/# chroot /sysroot
You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command.
sh-4.2# vi /etc/fstab
When you have completed the required changes/repairs, enable an SELinux relabel on reboot by touching the /.autorelabel file. This will ensure the SELinux contexts are set for any file changes made in emergency mode. Use the exit command to close the chroot environment, then again to reboot the instance.
sh-4.2# touch /.autorelabel sh-4.2# exit switch_root:/# exit
Ubuntu Recovery Mode
To enter Ubuntu recovery mode, hold the Shift key as the instance is reboots. If the GRUB boot menu does not appear, try again making sure you start holding the Shift key immediately after you click the Send CtrlAltDel button. If this doesn't work, press the ESC key multiple times right after clicking the CtrlAltDel button. When the GRUB boot menu appears, select Advanced options for Ubuntu and press Enter.
Then select Ubuntu, with Linux ... (recovery mode).
The Recovery Menu provides various recovery tools/options. To access the root shell prompt, you can select the root option and press Enter.
The root shell prompt has your instance root filesystem mounted in read-only mode.
You can remount the filesytem in read-write mode as follows.
root:~# mount -o remount,rw /
You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command.
root:~# vi /etc/fstab
When you have completed the required changes/repairs, use the exit command to close the shell and return to the Recovery Menu.
root:~# exit
In the Recovery Menu, select the resume option and press Enter to resume normal instance reboot. Press Enter again at the graphics drivers note.
OpenStack Rescue Mode
The OpenStack server rescue command can also be used to access your image in some cases. It boots from a fresh image of the image you originally used when you launched the instance, with a disk device configured for your image. You can use the disk device to mount your image and make the required configuration fixes. You will need to have the OpenStack server client installed locally.
Install the OpenStack command line client as described in the Manage instance via API Nectar Solutions article.
To put the server in rescue mode, first stop the instance.
openstack server stop <your instance id>
Check that the server has stopped using the server status command.
openstack server show <your instance id>
Once stopped, start the instance in rescue mode.
openstack server rescue <your instance id>
Check that the server is in rescue mode using the server status command. The server will be ready when vm_state is "rescued".
openstack server show <your instance id>
You should now be able to login with your Nectar SSH key file as usual.
ssh -i <your key file> <ec2-user|ubuntu>@<your instance ip address> ... $
Use the fdisk command to get the disk device for your instance.
$ sudo fdisk -l Disk /dev/vda: 2 GiB, 2147483648 bytes, 4194304 sectors ... Device Boot Start End Sectors Size Id Type /dev/vda1 * 2048 4192255 4190208 2G 83 Linux Disk /dev/vdb: 30 GiB, 32212254720 bytes, 62914560 sectors ... Device Boot Start End Sectors Size Id Type /dev/vdb1 * 2048 62914526 62912479 30G 83 Linux
In this example the required device is /dev/vdb1. You can mount the instance filesystem as follows.
$ sudo mount /dev/vdb1 /mnt
You can now go ahead and make the required changes/repairs, such as correcting problem entries in the fstab file or scanning the filesystem with the fsck command. Note that you need to prefix filenames with /mnt to access the instance filesystem.
$ sudo vi /mnt/etc/fstab
When done, logout and restart the instance in normal mode.
openstack server unrescue <your instance id>
