vCenter discovery errors. Basic troubleshooting.

Document created by dominic.grigg Expert on Aug 22, 2019Last modified by dominic.grigg Expert on Sep 16, 2019
Version 2Show Document
  • View in full screen mode

Even if a vCenter target has successfully validated there may be issues that later cause discovery of the environment to fail. 

 

If there is a discovery failure for a vCenter target notifications should be seen to this effect in the notifications centre.

 

This can also become apparent if the view of vCenter environment is out of date and for example 

vCenter VM moves fail because Turbonomic believes them to be on a different host to where they actually are, ie you may see the migration fail with :

 

<vm_name> is actually on host <host_name>. 

 

may also see action exceptions where actions cannot be executed: 

 

ERROR [sdk-op-12733] [ActionController] : <vCenter_target> : Action encountered exception

 

If there is a vCenter discovery failure in the catalina log for the vCenter in question there will be an ERROR for that vCenter with 'discovery failed'. 

 

This could be due to a network issue, a lack of heap memory on the Turbonomic instance, a credentials issue if these have been changed or a vCenter permissions issue if permissions for the Turbonomic user have been set below the vCenter level in vCenter.

 

-To check for network connectivity issues you can try running the following commands on the Turbonomic command line to ensure that you can reach the vCenter: 

 

ping [your vCenter target]
tracepath -p 80 [your vCenter target]
tracepath -p 433 [your vCenter target]

try curl -v -X GET -k "https://[vCenter address]/sdk/vimService?wsdl"

 

Connection timed out errors: 

 

Discovery 12124 of target <vCenter_target> timed out

java.util.concurrent.TimeoutException: Timing out while waiting for the response for message 12124 for 30000ms

 

Error during target <vCenter_target> discovery: Unable to get the DiscoveryResponse message. Connection timed out. 

 

--To check for network connectivity issues you can try running the following commands on the Turbonomic command line to ensure that you can reach the vCenter: 

 

ping [your vCenter target]
tracepath -p 80 [your vCenter target]
tracepath -p 433 [your vCenter target]

try curl -v -X GET -k "https://[vCenter address]/sdk/vimService?wsdl"

 

--Check memory resources on the Turbonoimic instance: 

 

-Check Java Garbage Collection in the catalina log: 

 

grep -i "allocation failure" catalina.out 

 

If allocation failure messages are seen then heap memory is no longer being reclaimed and it may be necessary to increase the memory of the instance and restart to resolve the issue. 

 

-You can also increase the connection timeout values for the vCenter probe if there is a slow connection:

 

In /srv/tomcat/data/config/sdk.probe.properties, we can try increase the connection timeouts.

1)

#probe.vCenter.target.connectTimeout=30000

#probe.vCenter.target.readTimeout=180000

to:

probe.vCenter.target.connectTimeout=300000

probe.vCenter.target.readTimeout=1800000


 

2) service tomcat restart

 

Permission denied errors: 

 

There errors will be seen if the Turbonomic user the vCenter was added with does not have sufficient permissions to discover or perform actions on the vCenter environment. This will probably be because the permissions for the Turbonomic user are not set at the vCenter level but have been set for individual clusters meaning that the permissions do not propagate down to child objects like datastores: 

ERROR: Login failed to <vCenter>  
Caused by: java.net.ConnectException: Permission denied (connect failed)

See GC Adding the required permissions to Operations Manager in vCenter 

For setting non-admin permissions if necessary see VirtualCenter Permissions Required for Turbonomic 

If there 

 

If there is an issue on the vCenter side in responding to our APIs you may also see http 500 internal server errors returned: 

 

ERROR [sdk-op-8] [WSClient] : post failed: java.io.IOException: Server returned HTTP response code: 500 for URL: https://<vCenter_target>/sdk

 

Unknown host exception: 

 

The java.net.UnknownHostException exception as in the errors below generally represents that the vCenter hostname cannot be resolved to an IP address. Check DNS is working correctly and as a test perhaps add the vCenter to the /etc/hosts file of the Turbonomic instance. 

 

ERROR [embedded-sdk-client-2] [VimSessionFactory] : Failed to get session key

java.lang.RuntimeException: java.rmi.RemoteException: VI SDK invoke exception : RetrieveProperties; nested exception is:

java.net.UnknownHostException: <vCenter_host_name>

 

Caused by: java.net.UnknownHostException: <vCenter_host_name>

 

Attachments

    Outcomes