Was it a disaster situation? A 0-day? or a data center fire? Nope, not a flashy problem or even a spectacular firestorm.
An inadvertent change of a Microsoft Server 2012 R2 VMware template’s NIC from VMXNET3 to E1000. Effectively reducing the NIC bandwidth from 10Gb to 1Gb as well as other benefits of using the VMXNET3 driver over the emulated driver of the E1000. This went unnoticed and affected ~50 VM’s.
A coworker was troubleshooting backup slowness issues and brought his findings to me.
I used the below PowerCLI command on each vCenter to view and exported to CSV any VM whose NIC was not equal to “Vmxnet3”. Thanks to @LucD22 from a 2012 post https://communities.vmware.com/thread/395849
Connect-VIServer vserver.yourdomain.com Get-VM | Get-NetworkAdapter | Where-object {$_.Type -ne "Vmxnet3"} | Select @{N="VM";E={$_.Parent.Name}},Name,Type | export-Csv c:\temp\Network_Interface.csv -NoTypeInformation
The resulting CSV file looks like the below. Some VM’s needed to stay on e1000 as they were Linux appliances, but approximately 50 Windows VM’s needed to be changed.
The resolution didn’t turn out to be as easy as just changing the Network Interface to VMXnet3. A PowerCLI one liner would have finished the job in a couple of minutes, but it was not to be.
Changing the VMs existing Network Interface from E1000 to VMXnet3 was not the problem. The VM Operating System was having the issue, 2012R2 didn’t recognize the NIC as having changed. Even after a reboot and scanning for new hardware changes in Device Manager.
Since I was going to have to reconfigure the static IP addressing information anyway, I ended up just deleting the existing NIC’s and adding a new one. Booting the VM and re-configuring the IP information. I had maintenance down time at night so the ~50 VM’s were changed starting with higher priority servers. If I had more VM’s to correct I would have looked for a fully automated solution.
Templates changes should be reported on and should be checked periodically for adherence to corporate standards as well as industry best practices.
VMXnet3 vs E1000e - https://rickardnobel.se/vmxnet3-vs-e1000e-and-e1000-part-1