A silent killer

Was it a disaster situation? A 0-day? or a data center fire? Nope, not a flashy problem or even a spectacular firestorm.

It was a simple change

An inadvertent change of a Microsoft Server 2012 R2 VMware template’s NIC from VMXNET3 to E1000. Effectively reducing the NIC bandwidth from 10Gb to 1Gb as well as other benefits of using the VMXNET3 driver over the emulated driver of the E1000. This went unnoticed and affected ~50 VMs.


A coworker was troubleshooting backup slowness issues and brought his findings to me.

I used the below PowerCLI command on each vCenter to view and exported to CSV any VM whose NIC was not equal to Vmxnet3”. Thanks to @LucD22 from a 2012 post https://communities.vmware.com/thread/395849

Connect-VIServer vserver.yourdomain.com Get-VM | Get-NetworkAdapter | Where-object {$_.Type -ne "Vmxnet3"} | Select @{N="VM";E={$_.Parent.Name}},Name,Type | export-Csv  c:\temp\Network_Interface.csv -NoTypeInformation


The resulting CSV file looks like the below. Some VMs needed to stay on e1000 as they were Linux appliances, but approximately 50 Windows VMs needed to be changed.

PowerCLI CSV Results


The resolution didn’t turn out to be as easy as just changing the Network Interface to VMXnet3. A PowerCLI one liner would have finished the job in a couple of minutes, but it was not to be.

Changing the VMs existing Network Interface from E1000 to VMXnet3 was not the problem. The VM Operating System was having the issue, 2012R2 didn’t recognize the NIC as having changed. Even after a reboot and scanning for new hardware changes in Device Manager.

Since I was going to have to reconfigure the static IP addressing information anyway, I ended up just deleting the existing NICs and adding a new one. Booting the VM and re-configuring the IP information. I had maintenance down time at night so the ~50 VMs were changed starting with higher priority servers. If I had more VMs to correct I would have looked for a fully automated solution.

Lessons Learned

Templates changes should be reported on and should be checked periodically for adherence to corporate standards as well as industry best practices.


VMXnet3 vs E1000e - https://rickardnobel.se/vmxnet3-vs-e1000e-and-e1000-part-1

What’s faster - https://www.ntpro.nl/blog/archives/1478-Whats-faster-E1000-or-VMXNET3-lets-see-what-PassMarks-PerformanceTest-says.html

Up next 0035-forensic_lunch_with_david_cowen 0037-how_to_get_the_last_boot_time
Latest posts 0104-change-synology-password-cli 0105-free-git-ebook 0103-using-brew-bundle-to-backup-and-restore-mac-app-store-and-brew-apps Update macOS with an all in one alias Mac App Store Command Line Interface 0100-macos-softwareupdate-cli Markdown Crash Course Video What’s New with Fusion and Workstation [HCP1833] File IO in Python Overview of vSphere 7 Video 0095-what_is_iso Migrating Website HTML details Tag Microsoft RD Client iOS App 0091-create_vmware_esxi_usb_install_media Intel NUC Lab Hosts Hardware Setup 0089-installng_microsoft_sql_2016 Installing the First Two Domain Controllers in the VMware ESXi 6.7 Lab Environment 0088-deploy_vcsa_in_lab Macchanger Utility, and Usage Install a Kali Linux VM in a VMware ESXi 6.7 Environment Install Ubuntu 18.04 Virtual Machine in a VMware ESXi 6.7 Environment Install xRDP on Ubuntu 18.04 Install Chromium on Ubuntu 18.04 Install OpenSSH on Ubuntu 18.04 Install VMware Workstation 14 on Ubuntu 18.04 Nested ESXi server Laboratorium Rattus The Animal Within Who is Veeam? RSAC OnDemand Videos 0075-create_win10_sandbox_vm