Unable to vMotion, enable SSH – “Can’t Fork” or “Heap globalCartel-1 already at its maximum size”

There has been a new trending issue with the HP AMS agent that causes vMotions to fail, enabling SSH on the host to fail, and overall host management functionality is lost. If you attempt to power on a virtual machine you may see an error message saying “VMK_NO_MEMORY.” When attempting to open a console you may see the error “Unable to contact the MKS: Could not connect to pipe\\.\pipe\vmware-authpipe.”

When you log into the ESXi shell you will recieve the error message “can’t fork” and if you Press Alt + F12 to see a tail of the vmkernel logs you will see the following error message.

"WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand."

This issue is caused by the hp-ams service and the versions affected are below.

hp-ams 500.9.6.0-12.434156
hp-ams-550.9.6.0-12.1198610
hp-ams 500.10.0.0-18.434156
hp-ams-550.10.0.0-18.1198610

To see if you are affected, you can run the command

# esxcli software vib list | grep ams

You can workaround this issue using the following steps.

Stop the HP-AMS service

# /etc/init.d/hp-ams.sh stop

Remove the HP-AMS vib

# esxcli software vib remove -n hp-ams

 
Once that is done you can upgrade/install HP AMS version 10.0.1 or newer where the issue no longer exists. I have provided a link to these downloads below.
 
ESXi 5.1

ESXi 5.5
 
If you have a host that is exhibiting these behaviors and you are unable to stop the service you can power off non-critical virtual machines to clear resources, then attempt to stop the hp ams service. If all else fails, you will have to reboot the host during a maintenance window.

Hope this helps!

 

Posted by:

Sean Whitney

Leave A Comment

Your email address will not be published. Required fields are marked (required):

Back to Top