VXLAN vib agent installations fails immediately with the error “Agent VIB module not installed”

 
I ran into an issue the other day in a new NSX lab deployment I was building out. Whenever I tried to prepare the ESXi hosts for VXLAN the install would immediately fail with no real good error messages; tasks and events displayed “Cannot complete the operation. See the event log for details.” Task: XXX eam.agent.enable.label not found XXX.”After digging a little bit deeper, I was able to find the following error message in my NSX Manager log.
 

 
2015-06-28T20:02:58.323-06:00 | ERROR | agent-4 | AuditedJob.java | 75 | JOB FAILED: [#2025773242] EnableDisableAgentJob(AgentImpl(ID:'Agent:7cd8e40b-7892-4755-977f-b847b4d35eae:bb36baa5-dd5d-46cc-b80c-eb38df42ff3c')), com.vmware.eam.EamException: VibInstallationFailed
2015-06-28T20:02:58.324-06:00 | DEBUG | agent-4 | AuditedJob.java | 77 | Cause: 
com.vmware.eam.EamException: VibInstallationFailed
	at com.vmware.eam.job.EnableDisableAgentJob.runWorkflow(EnableDisableAgentJob.java:88)
	at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:89)
	at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:37)
	at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
	at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2015-06-28T20:02:58.325-06:00 |  INFO | host-20-0 | VcComputeResource.java | 505 | VcClusterComputeResource(ID: domain-c7) setting required agent count in VC to 0
2015-06-28T20:02:58.368-06:00 | ERROR | host-20-0 | AuditedJob.java | 75 | JOB FAILED: [#9241039] VibJob(VcHostSystem(ID: host-20)), com.vmware.eam.EamException: Failed to complete VIB task
2015-06-28T20:02:58.368-06:00 | DEBUG | host-20-0 | AuditedJob.java | 77 | Cause: 
com.vmware.eam.EamException: Failed to complete VIB task
	at com.vmware.eam.job.VibJob.rethrow(VibJob.java:183)
	at com.vmware.eam.job.VibJob.call(VibJob.java:176)
	at com.vmware.eam.job.VibJob.call(VibJob.java:65)
	at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
	at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused: connect
	at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:251)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:54)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)
	at com.sun.proxy.$Proxy97.retrieveVcIntegrityContent(Unknown Source)
	at com.vmware.eam.vum.VumConnection.connect(VumConnection.java:86)
	at com.vmware.eam.vum.VumConnection.checkConnection(VumConnection.java:169)
	at com.vmware.eam.vum.VumConnection.getVcIntegrityContent(VumConnection.java:193)
	at com.vmware.eam.job.VibJob.validateLiveVibInstall(VibJob.java:333)
	at com.vmware.eam.job.VibJob.installVib(VibJob.java:271)
	at com.vmware.eam.job.VibJob.updateVibs(VibJob.java:226)
	at com.vmware.eam.job.VibJob.call(VibJob.java:158)
	... 6 more
Caused by: java.net.ConnectException: Connection refused: connect
	at java.net.DualStackPlainSocketImpl.connect0(Native Method)
	at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
	at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.connectSocket(VlsiSslSocketFactory.java:48)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
	at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
	at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
	... 21 more

 
This error message still didn’t really point me anywhere other than I was receiving some type of connection refused error messages. I assumed this was an EAM error but I wasn’t seeing anything in the eam.log on vCenter Server, and it didn’t appear that the esxupdate.log was even mentioning the event which means this task was never making it to the host. One thing I remembered though, was that the EAM service was utilized by Update Manager to push out patches. When I went to services I noticed the VMware Update Manager Service was stopped. As a long shot, I decided to start it and retry my NSX vib installs. To my surprise, the cluster was prepared quickly without any issues. This lead me to the conclusion that the EAM service will rely on VMWare Update Manager to push VIBs out when the service is installed, however if Update Manager is not installed, EAM will push out the VIBs without VUM.