I recently sat in on an internal VNX (and CLARiiON) performance crash course that was put together to help our new hires get up to speed.  Once of the things that stuck out to me was the subject of iSCSI and how it works with host TCP delayed acknowledgement (Delayed ACK).

Background Information

So what is delayed ACK?  As part of TCP, for every packet that is sent to a destination server, that server must send some sort of acknowledgement back to the source server.  This way the source server knows the information was successfully transmitted.  This adds a good amount of overhead, so in an effort to improve performance, TCP Delayed Acknowledgement (RFC 1122) was created which allows a destination server to respond to every 2nd packet instead.  This has become so popular, that support for delayed ACK is enabled by default in many popular client operating systems including Microsoft Windows and VMware ESX/i.

The problem with this is that many storage arrays do not support delayed ACK for one reason or another (usualy has to do with chipset drivers).  What happens in this case is that the array will send a packet, it will then wait for an acknowledgement before sending a 2nd packet.  Meanwhile, the host is waiting for a 2nd packet before sending an acknowledgement.  This standoff between the array and the host will last until the acknowledgement timeout (usually around 200ms) before continuing on.  This wreaks havoc on performance if every packet has to wait 200 milliseconds before sending another.  So if you’ve setup iSCSI and you are immediately seeing a performance issue, check your hosts to see if Delayed ACK is enabled, and turn it off to see if performance improves.

Disabling Delayed ACK in Microsoft Windows

In Microsoft Windows operating systems, you can simply set the TcpAckFrequency registry value to 1.  More information can be found Microsoft kb 328890.  On a side note, I found that if the registry value is missing, you can create it in the path specified in the kb and reboot the host.

Disabling Delayed ACK in VMware ESX and ESXi

VMware has created KB 1002598 to address this as well.  This adjustment is made per adapter instance and you can change this setting on a discovery address, a target, or (in my case) globally.  Once you’ve made your change, reboot the host and enjoy the performance boost.

I hope you’ve found this information useful.  It may not solve your iSCSI performance problem, but it is a good place to start.