VXLAN vib agent installations fails immediately with the error “Agent VIB module not installed”
I ran into an issue the other day in a new NSX lab deployment I was building out. Whenever I tried to prepare the ESXi hosts for VXLAN the install would immediately fail with no real good error messages; tasks and events displayed “Cannot complete the operation. See the event log for details.” Task: XXX eam.agent.enable.label not found XXX.”After digging a little bit deeper, I was able to find the following error message in my NSX Manager log.
2015-06-28T20:02:58.323-06:00 | ERROR | agent-4 | AuditedJob.java | 75 | JOB FAILED: [#2025773242] EnableDisableAgentJob(AgentImpl(ID:'Agent:7cd8e40b-7892-4755-977f-b847b4d35eae:bb36baa5-dd5d-46cc-b80c-eb38df42ff3c')), com.vmware.eam.EamException: VibInstallationFailed 2015-06-28T20:02:58.324-06:00 | DEBUG | agent-4 | AuditedJob.java | 77 | Cause: com.vmware.eam.EamException: VibInstallationFailed at com.vmware.eam.job.EnableDisableAgentJob.runWorkflow(EnableDisableAgentJob.java:88) at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:89) at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:37) at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35) at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-06-28T20:02:58.325-06:00 | INFO | host-20-0 | VcComputeResource.java | 505 | VcClusterComputeResource(ID: domain-c7) setting required agent count in VC to 0 2015-06-28T20:02:58.368-06:00 | ERROR | host-20-0 | AuditedJob.java | 75 | JOB FAILED: [#9241039] VibJob(VcHostSystem(ID: host-20)), com.vmware.eam.EamException: Failed to complete VIB task 2015-06-28T20:02:58.368-06:00 | DEBUG | host-20-0 | AuditedJob.java | 77 | Cause: com.vmware.eam.EamException: Failed to complete VIB task at com.vmware.eam.job.VibJob.rethrow(VibJob.java:183) at com.vmware.eam.job.VibJob.call(VibJob.java:176) at com.vmware.eam.job.VibJob.call(VibJob.java:65) at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35) at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.ConnectException: Connection refused: connect at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:251) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:54) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186) at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308) at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182) at com.sun.proxy.$Proxy97.retrieveVcIntegrityContent(Unknown Source) at com.vmware.eam.vum.VumConnection.connect(VumConnection.java:86) at com.vmware.eam.vum.VumConnection.checkConnection(VumConnection.java:169) at com.vmware.eam.vum.VumConnection.getVcIntegrityContent(VumConnection.java:193) at com.vmware.eam.job.VibJob.validateLiveVibInstall(VibJob.java:333) at com.vmware.eam.job.VibJob.installVib(VibJob.java:271) at com.vmware.eam.job.VibJob.updateVibs(VibJob.java:226) at com.vmware.eam.job.VibJob.call(VibJob.java:158) ... 6 more Caused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.connectSocket(VlsiSslSocketFactory.java:48) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ... 21 more
This error message still didn’t really point me anywhere other than I was receiving some type of connection refused error messages. I assumed this was an EAM error but I wasn’t seeing anything in the eam.log on vCenter Server, and it didn’t appear that the esxupdate.log was even mentioning the event which means this task was never making it to the host. One thing I remembered though, was that the EAM service was utilized by Update Manager to push out patches. When I went to services I noticed the VMware Update Manager Service was stopped. As a long shot, I decided to start it and retry my NSX vib installs. To my surprise, the cluster was prepared quickly without any issues. This lead me to the conclusion that the EAM service will rely on VMWare Update Manager to push VIBs out when the service is installed, however if Update Manager is not installed, EAM will push out the VIBs without VUM.
4 Comments
Sean, I’ve also seen this exact error when DNS is not properly configured on a host.
Hi Mike,
That’s correct, same error but logs will usually say unable to contact host.
Sean
i corrected DNS on host and it worked
Thanks guys! DNS entry resolved this for me …..