Unable to deploy NSX VIBs after updating certificates in vSphere 6.0
I ran into an issue the other day while helping a customer install the NSX agents on his ESXi hosts. Whenver we tried to deploy the agents to the host, we would get a generic error in the vSphere Web Client. To troubleshoot the issue, we first navigated to Home -> Administration -> vCenter Server Extensions and double clicked vSphere ESX Agent Manager.
By clicking the Manage tab we saw no ESX Agencies / an error similar to:
“Error while creating eam agency for deployment”
In the NSX Manager log you see the following:
show manager log 2015-05-29 17:23:42.120 GMT ERROR taskScheduler-15 InstallTask:190 - error while creating eam agency for deployment com.vmware.vim.binding.eam.fault.NoConnectionToVCenter: inherited from com.vmware.vim.binding.eam.fault.EamRuntimeFault: inherited from com.vmware.vim.binding.eam.fault.NoConnectionToVCenter
This pointed to a problem with the ESX Agent Manager service on vCenter Server. After checking the EAM logs under
Note: The EAM logs are located here:
Appliance: /var/log/vmware/eam/eam.log Windows: C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log
eam.log Connecting to vCenter as com.vmware.vim.eam extension Connecting to https://:8089/sdk/vimService via vCenter proxy http://localhost:80 HealtStatus request's token subject name: machine-7502fb4c-3521-48c7-93ed-3d1865e0fff1, subject domain: vsphere.local Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password. | WARN | eam-0 | VcListener.java | 134 | Trying to recover from error (vim.fault.InvalidLogin) { faultCause = null, faultMessage = null } at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at java.lang.Class.newInstance(Unknown Source) at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173) at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext. (ComplexStackContext.java:31) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89) at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84) at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41) at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112) at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273) at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182) at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source) at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:171) at com.vmware.eam.vc.VcListener.login(VcListener.java:149) at com.vmware.eam.vc.VcListener.main(VcListener.java:129) at com.vmware.eam.vc.VcListener.call(VcListener.java:111) at com.vmware.eam.vc.VcListener.call(VcListener.java:60) at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35) at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) VcListener.java | 121 | Retrying in 10
The only thing we did to induce this error message was to change the vCenter Server certificates. It appears that once you do this, the EAM service is no longer able to communicate properly with vCenter Server. To resolve this issue, you can follow the steps below.
Windows
Open a command prompt as administrator and run the following commands. The first two will retrieve the vpxd-extension solution user cert and key; the last commands will update the EAM certificate with vCenter Server.
# "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.crt # "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.key # cd C:\Program Files\VMware\vCenter Server\vpxd\scripts\ # "%VMWARE_PYTHON_BIN%" updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificates\vpxd-extension.crt -k C:\Certificates\vpxd-extension.key -s localhost -u Administrator@vsphere.local
Provide your administrator@vsphere.local password when prompted.
Appliance
Log into the vCenter Server appliance via SSH and run the following commands.
# shell.set --enabled true # mkdir /certificate # /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt # /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
Provide your administrator@vsphere.local password when prompted.
3 Comments
Also for the appliance you will need to run the python script:
# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u administrator@vsphere.local
Password to connect to VC server for user=”administrator@vsphere.local”:
2016-03-11T08:55:19.226Z Updating certificate for “com.vmware.vim.eam” extension
2016-03-11T08:55:19.331Z Successfully updated certificate for “com.vmware.vim.eam” extension
2016-03-11T08:55:19.434Z Verified login to vCenter Server using certificate=”/certificate/vpxd-extension.crt” is successful
#
Lars
I’ve had to perform this fix several times for many customers. I went to do it again right now and discovered the VMware KB website is down for maintenance. Thankfully you’ve capture the steps here and saved me having to reschedule this. Thanks!
Happy to help, Hamish!