One of the theories I have, and the folks I work with, is that VMs that generate a lot of network traffic between each other will perform better if they are on the same host than on two separate hosts. So let's say we have 2 ESX hosts, the VMs are using the VMXNET3 (10gb) NICs, and we have a 1gb network switches. The thought is that the VMs will achieve better network throughput if they are on the same host because they will not use the physical uplink to the network switch, the network traffic is handled by the vSwitch instead.
So in a HA configuration where the 2 hosts are equal hardware wise, is it better to leave all of the VMs on 1 host if the "Active" host can handle the load? Or is it better to load balance the VMs on the two hosts to take advantage of the unused hardware resources?