Hyper-V High Ping Latency

Had an interesting issue today, an insanely slow, brand new server….. or so I thought. First a bit of background on the client, they are fairly large, with over 400 client access devices to maintain, not including server, network equipment etc. to support this the client has 3 servers, one purchased a year and the oldest one thrown out, keeping all devices in warranty and with modern powerful equipment to keep things running, in addition to this there are two other servers that are replaced every three years, these are treated differently as these are speciality servers, and only do one task.

So, building a new server for a client, in this case a Dell R520 with Server 2008R2 running as a Hypervisor, nothing special in that, I do this exceedingly regularly so it has become more of a routine build for me. What got me with this one though was during testing I was getting insanely high ping latency, not only to the virtual machines from the network and vice versa, but also from the hypervisor to the machines and vice versa. Pings to other virtual machines on others server, on different LAN segments were all responding normally in <1MS

My first thought was there was something wrong with the virtual machines and that I had butchered something in the migration, but as they worked on other hypervisors without delays, that knocked that one on the head. Then I thought network location issue, but that does not make any sense due the fact that pinging from a hypervisor to guest does not go across a physical network, so it has to be the brand new server.

Ok, so what’s new about this server, well its got newer Processors, greater memory, faster HDD’s with larger capacity’s, basically it was more a case of what wasn’t different to the last server. Not going to go through the whole process of troubleshooting, but basically it was to do with the NIC’s, fine now what about them is it. After trial and error, and of course every techs most important tool, Google I came across the issue what is it…

 

THE ISSUE IS VMQ or Virtual Machine Queuing inside the Broadcom NIC drivers as shown below, disable this and the issue clears instantly

Advanced NIC Properties showing Virtual Machine Queue option
Advanced NIC Properties showing Virtual Machine Queue option

 

Pings and other indicators are now back down to <1ms which is what I expected to see in the first place.

Hardware effected by this was as follows

DELL R520
Server 2008R2 Enterprise
Broadcom Quad Port NIC
Broadcom Driver 15.4.0.17 dated 4th of September 2012, as downloaded from the DELL site on the 31st of January 2013

Going to put this on my to check list in future

 

UPDATE 11th February 2013:

Dennis over at Flexecom has found the same thing in this posting (http://www.flexecom.com/high-ping-latency-in-hyper-v-virtual-machines/), posted on the 10th of December, wish I had found it before so I did not have to troubleshoot this myself, none the less he has more information on how VMQ’s are MEANT to work, interestingly although it is a different manufacturer, the NIC is the same, as is the driver version although the reported release date of the driver is different, so currently the problem seems to exist with BROADCOM NIC’s and specifically using driver revision 15.4.0.17. Perhaps we could get Broadcom to turn this off by default, then if desired the server admin could turn it on.

 

Written By Justin

15 Comments on “Hyper-V High Ping Latency

  1. Chris Reply

    April 5, 2013 at 2:16

    Thanks for the tip! Couldn’t get it figured out after I had to break the Teaming on the BASP and my hyperv performance dropped like a rock.

    • ShadowPeo Reply

      April 6, 2013 at 8:23

      Happy it helped mate,

      Justin

  2. AnaLi Reply

    June 3, 2013 at 7:31

    Thank you! resolved for me this trick!

    • ShadowPeo Reply

      December 27, 2013 at 9:12

      Happy to have helped

  3. Abhirup Reply

    December 27, 2013 at 8:50

    Thanks a ton Justin… Was struggling badly with this issue for the past few weeks… Now its resolved…

    • ShadowPeo Reply

      December 27, 2013 at 9:12

      Happy to have helped

  4. Luke Brynycz Reply

    February 4, 2014 at 1:59

    Thanks guys! I was scratching my head at this one also. I was playing with ports on switches and all sorts.

    • ShadowPeo Reply

      February 11, 2014 at 2:19

      Happy to Help

  5. Shahbaz Khan Reply

    April 16, 2014 at 1:10

    Thank you soo much mate…really saved alot of time for this one
    Cheers!

  6. Roquefort Reply

    September 9, 2014 at 5:41

    Thank you! This worked perfectly.

  7. jay Reply

    May 7, 2015 at 7:55

    thanks very much for taking the effort to write this up – we just came across it ourselves and never seen it before despite having a heap of hyper-v servers (probably a good 30 or so)

  8. venhow Reply

    October 22, 2015 at 2:34

    coool,,,man..tks…

  9. Felipe Lucena Reply

    April 1, 2016 at 12:32

    Your tip across the continent. I am in Recife / Brazil. That your tip saved my day!

    Hugs.

  10. saran Reply

    July 23, 2016 at 2:36

    Thanks, it helps to solve the problem.

Leave a Reply to jay Cancel reply

Your email address will not be published. Required fields are marked *

one × 4 =