Unable to deploy NSX VIBs after updating certificates in vSphere 6.0

I ran into an issue the other day while helping a customer install the NSX agents on his ESXi hosts. Whenver we tried to deploy the agents to the host, we would get a generic error in the vSphere Web Client. To troubleshoot the issue, we first navigated to Home -> Administration -> vCenter Server Extensions and double clicked vSphere ESX Agent Manager.
 
1
 
By clicking the Manage tab we saw no ESX Agencies / an error similar to:
 
“Error while creating eam agency for deployment”
 
In the NSX Manager log you see the following:
 

show manager log
2015-05-29 17:23:42.120 GMT ERROR taskScheduler-15 InstallTask:190 - error while creating eam agency for deployment 
com.vmware.vim.binding.eam.fault.NoConnectionToVCenter:
inherited from com.vmware.vim.binding.eam.fault.EamRuntimeFault:
inherited from com.vmware.vim.binding.eam.fault.NoConnectionToVCenter

 
This pointed to a problem with the ESX Agent Manager service on vCenter Server. After checking the EAM logs under we were able to find the following error message.
 
Note: The EAM logs are located here:

Appliance: /var/log/vmware/eam/eam.log 
Windows: C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log

 

eam.log
Connecting to vCenter as com.vmware.vim.eam extension
Connecting to https://:8089/sdk/vimService via vCenter proxy http://localhost:80
HealtStatus request's token subject name: machine-7502fb4c-3521-48c7-93ed-3d1865e0fff1, subject domain: vsphere.local
Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
 |  WARN | eam-0 | VcListener.java | 134 | Trying to recover from error
(vim.fault.InvalidLogin) {
  faultCause = null,
  faultMessage = null
}
 at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
 at java.lang.reflect.Constructor.newInstance(Unknown Source)
 at java.lang.Class.newInstance(Unknown Source)
 at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
 at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
 at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.(ComplexStackContext.java:31)
 at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)
 at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)
 at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)
 at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)
 at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
 at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)
 at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)
 at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
 at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)
 at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
 at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)
 at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77)
 at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
 at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
 at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)
 at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)
 at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)
 at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source)
 at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:171)
 at com.vmware.eam.vc.VcListener.login(VcListener.java:149)
 at com.vmware.eam.vc.VcListener.main(VcListener.java:129)
 at com.vmware.eam.vc.VcListener.call(VcListener.java:111)
 at com.vmware.eam.vc.VcListener.call(VcListener.java:60)
 at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
 at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
VcListener.java | 121 | Retrying in 10

 
The only thing we did to induce this error message was to change the vCenter Server certificates. It appears that once you do this, the EAM service is no longer able to communicate properly with vCenter Server. To resolve this issue, you can follow the steps below.
 

Windows

 
Open a command prompt as administrator and run the following commands. The first two will retrieve the vpxd-extension solution user cert and key; the last commands will update the EAM certificate with vCenter Server.

# "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.crt

# "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.key

# cd C:\Program Files\VMware\vCenter Server\vpxd\scripts\

# "%VMWARE_PYTHON_BIN%" updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s localhost -u Administrator@vsphere.local

 
Provide your administrator@vsphere.local password when prompted.
 

Appliance

 
Log into the vCenter Server appliance via SSH and run the following commands.
 

# shell.set --enabled true

# mkdir /certificate

# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

 
Provide your administrator@vsphere.local password when prompted.
 

Posted by:

Sean Whitney

3 Comments

  1. Lars Troen -  March 11, 2016 - 5:00 am 438

    Also for the appliance you will need to run the python script:

    # python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u administrator@vsphere.local
    Password to connect to VC server for user=”administrator@vsphere.local”:
    2016-03-11T08:55:19.226Z Updating certificate for “com.vmware.vim.eam” extension
    2016-03-11T08:55:19.331Z Successfully updated certificate for “com.vmware.vim.eam” extension
    2016-03-11T08:55:19.434Z Verified login to vCenter Server using certificate=”/certificate/vpxd-extension.crt” is successful
    #

    Lars

    Reply
  2. Hamish -  May 19, 2017 - 11:03 pm 589

    I’ve had to perform this fix several times for many customers. I went to do it again right now and discovered the VMware KB website is down for maintenance. Thankfully you’ve capture the steps here and saved me having to reschedule this. Thanks!

    Reply
    • Sean Whitney -  November 4, 2017 - 10:20 am 619

      Happy to help, Hamish!

      Reply

Leave A Comment

Your email address will not be published. Required fields are marked (required):

Back to Top