XenServer – Pool Master Recovery (The Missing Part 1 to XenServer Hosts in Halted Mode)

In July of 2012 I wrote a “part 2” regarding XenServer Hosts in halted mode — however I seem to have misplaced part 1 – which I’ve rewritten after having to need to reference these steps again recently.

There are several events which can cause a XenServer Pool to become corrupt. In a recent instance of mine, the pool master was unable to communicate with the HA storage repository (SR) and fenced. I also had another instance where several shutdown unexpectedly, and the pool master was among them. Here are the steps I performed to recover the Pool Master.

  1. Work on recovering the pool, elect the server you want to become the master, and on that box run “xe pool-emergency-transition-to-master”
  2. Once that is completed, on the newly elected/transitioned master, run “xe pool-recover-slaves”
  3. Once that is complete, you should be able to run “xe host-list” and see all of your hosts listed


Based in part on information from: XenServer System Recovery Guide

Hung VM, unable to force reboot/shutdown

I have been working with a few vendor provided VM’s which run Linux. For some reason this specific set of Linux VMs do not properly respond when issuing reboot or shutdown commands when they VMs are hung. This is even true of force-shutdown. The following process works great for virtual servers that are non-responsive in a XenServer environment, after normal reboot/shutdown attempts have failed.

  1. “xe vm-list name-label={vm logical name}” to get the uuid of the VM that is hung
  2. “list_domains” to list the domain uuid’s so you can determine the domain # of the VM above by matching the uuids from this output with the uuid for your VM from the previous command.
  3. “/opt/xensource/debug/destroy_domain -domid XX” where XX is the domain number from the previous command
  4. “xe vm-reboot name-label={vm logical name} –force”



Based in part on information from: http://www.r2dtop.com/xenserver-6-virtual-machine-crash-and-hang-issue/


XenServer: Changing management adapter in pool

After going through several rounds of problems to move a management adapter for a xenserver pool, I have found the following working process. However, it is because of this processes that Citrix makes very clear that you should configure it properly in the first place, and if you need to make changes post-installation, to make them BEFORE you join it to a pool… Also you must change the subnet when changing interfaces. Even if you need to move it to a temporary, non-existant IP address space, and then move it back to the correct IP address space after you are on the correct network interface.

However, lets say you have a pool in production and you need to make the change…

  1. Perform a metadata backup and back up your virtual machines before performing the rest of this procedure.
  2. Disable High Availability from XenCenter, if enabled.
  3. Disable external authentication (Active Director)
  4. Log on to a pool member from the physical console and change the management interface IP address
  5. From the xsconsole, go to Network and Management Interface > Configure Management Interface.
    1. Note: xsconsole freezes when the change is applied. You can use the key sequence CTRL+Z to gain access to the command prompt to run step 4 below. Then, use the command fg %1 to return to xsconsole and exit cleanly.
  6. From the CLI: use the following command: xe pif-reconfigure-ip uuid= IP= gateway= netmask= DNS= mode=
  7. To locate the correct PIF uuid for pif-reconfigure command, use the following command: xe pif-list params=uuid,host-name-label,device,management
  8. From the CLI, run the following command: xe-toolstack-restart
  9. The server enters the emergency mode. Verify that the server is using the new IP address. You can ping it from another host. Try a Secure Shell connection to it, or use the ifconfig command. Verify that the server is in emergency mode by running xe host-is-in-emergency-mode from the CLI. You should get True as the output.
  10. Repeat steps 3 and 4 on each of the pool members.
  11. Change the management interface IP address on the pool master using step 3 above.
  12. Run the following command on the pool master: xe-toolstack-restart
    From the CLI, on each of the pool members, run xe pool-emergency-reset-master master-address=IP_OF_THE_MASTER.
  14. Verify the correct status of the pool. Connect with XenCenter to the new master’s IP address and check everything from there.
  15. Re-enable High Availability and external authentication, if required

If during this process, any of your pool-slave hosts reboot and show missing management interface, and no network cards, please see our post over at: https://reddingitpro.wordpress.com/2012/04/07/xenserver-missing-network-cards-pool-member/

You can also view a video walk through of this process at: http://www.citrix.com/tv/#videos/4330

Adapted from CTX123477

XenServer: Missing network interface – pool member

I have encountered several times that after a shutdown and restart on a XenServer host when it is configured as a pool, sometimes the pool members come up with no management interface because there are no network cards shown. The main reason I have seen this is because the pool-master server changes its IP address – this could be something as simple as changing the IP address, a DHCP address change, or a change of the PIF (physical interface) used for management. In these cases, if the pool slave cannot find the master, it will go into emergency mode to protect the VMs. However, the problem this presents is that there are no network cards available on the slave, no management interface, and the VM’s which were running on that server (even if you’re using shared storage) are unavailable.

The resolution is very quick and simple…

First, verify that you are in emergency mode by running the following from the command line interface on the pool slave host “xe host-is-in-emergency-mode” – – if it returns TRUE then read on, if it returns FALSE then this will not resolve your problem.

Next, verify what the IP address is of your running POOL MASTER (this assumes your POOL MASTER is still running, otherwise you will need to perform an emergency transition)….

On the pool slave, run the following “xe pool-emergency-reset-master master-address=xxx.xxx.xxx.xxx” — where xxx.xxx.xxx.xxx is the IP address of your working pool master…

Upon success it will notify you it will make the change within 10 seconds….

After 10 seconds, run “xe host-is-in-emergency-mode” — if it returns false you should be all set. You may need to refresh (disconnect/reconnect) to the pool in XenCenter.

If your pool-master is unavailable, or all of your hosts are showing no network adapter, then you will need to transfer the master role to one of the servers, from the command line, run: “xe pool-emergency-transition-to-master”

That will make this host the new pool master. Return to the menu system “xsconsole” and document that IP address as the pool master, and then continue the documentation above.


Powered by WordPress.com.

Up ↑