Unrecoverable error during 5.5 restore (from failed 5.6)
This weekend I decided to perform the upgrade of our 2 XenServer 5.5 servers in a farm configuration to 5.6 FP1. However I found conflicting information on how to perform the actual upgrade. The mistake I made was to put the server into maintenance mode before shutting it down. When performing the upgrade you must keep the pool master in normal mode, with all VMs migrated off of it, and then shut it down, which will place the farm into a recovery mode. While in this mode you are supposed to perform the implace upgrade in a rolling style. I miss read that step. So instead I ran the upgrade with the pool master in maintenance mode (thus it was no longer the true pool master as it nominated another server to be the master). Well it let me perform the upgrade, and everything appeared to be working fine. The server rebooted and I was greeted by the regular XSConsole. However I noticed two things:
1) XenCenter still saw the server as offline;
2) XSConsole showed that there were no network interfaces (NO NICS).
After researching the issue, I discovered it was caused by an improper upgrade, but no fear there is a build in restore option. Simply insert the upgrade CD and reboot… It will prompt with a restore option. And it was working great until about 95% where it errored out saying:
“Installer only supports having a single kernel of each type installed. Found 2 of kernel-xen”
Apparently if you have any prior backups on the server, plus the one made during the upgrade, the restore will fail. I found a Citrix Forum post http://forums.citrix.com/message.jspa?messageID=1521356 which described by specific situtation and I attempted the recovery to no success. Only having mild Linux experience it took be a while to discover what I was missing from that forum post since I am a Microsoft guy. Here is the actual steps for a windows guy:
1) Reboot the server with the 5.6 upgrade CD
2) When prompted for advanced setup, press F2 (it will quickly auto select standard install if you aren’t watching)
3) It will prompt you for which advanced setup mode, type “shell” and press enter (no quotes)
4) Setup will continue and dump you to a command line
5) Type “vi /opt/xensource/installer/backend.py” and press enter (again, without the quotes)
6) You are now in the VI editor which is a pain, you can google for how to nagivate, but for the purposes of this, type “/kernel” – and press enter, repeat that until you see the line beginning with “assert len(out) == 1, “Installer only supports having a single kernel ”
7) with the cursor over that line, type dd (this should delete the entire line)
8) Then move the cursor over to “return out” and press “a” to enter into the append mode, change it to read “return out[-1]” – then press “esc” and then type “ZZ” (Case sensative).