Virtualization - Thulin' Around

VNX INYO MR1–The Future is Now

By Sean | February 25, 2013 - 7:55 AM |February 28, 2013 DellEMC, VMware, VNX

With a new year, comes a HUGE update to the VNX family. As Chad Sakac reveled earlier in the year, INYO was the code name for the VNX FILE OE 7.1 and BLOCK OE 05.32 code release that surfaced last year. Now, the time has come for a major update to the code, and with it some exciting new features.

VNX OE for FILE

On the FILE side, the biggest (and what I think is the most exciting) feature coming is support for SMB 3.0 and the VNX is the first array to support this. Back in October of 2012, Microsoft released it’s latest versions of the Windows Operating system (Windows 8 and Server 2012). With that came the latest enhancements to the SMB protocol (for more information, click here to read a great blog post by Microsoft). With this upgrade (and the use of the SMB 3.0 protocol) you get a much less disruptive failover which includes keeping the open state of a file and file lock. You will also notice enhanced throughput by being able to take advantage of the Multi Path IO over SMB 3.0 without the need to configure LACP or EtherChannel.

VNX OE for BLOCK

On the BLOCK side, the VNX gains support for ODX support and the ability to Offloaded copies to the array. This cuts down on host CPU as well as SAN bandwidth as the transfers don’t leave the array. This is done by breaking down the copy into a series of tokens and passing them between the hosts while the data is passed between luns (as demonstrated in the chart below)

A couple of things to note. This does require an enabler, but you do not need to reboot the SP for that. You will have to reboot the host for that (it’s a limitation of Microsoft, not EMC). You will have to use Microsoft MPIO or the latest versions of PowerPath as well as an NTFS file system (with an allocation size of 8k or larger for better performance).

Also included with this release was several enhancements revolving around VAAI support on the VNX. Most this included the XCOPY fix as described in solution emc313487 as well as a big performance improvement to VAAI Fastclones. Chad has more on that subject here.

Unisphere Service Manager

Finally, there was also another enhancement that I wish I had when I was in tech support. Starting with the new version of Unisphere Service Manager (1.2.26.1.0068) you will now find a 1-click health check available after the main login. You may remember a previous blog post I did on how to run health checks on the VNX. Now you can run a single check to verify the health of your array (BLOCK, FILE, or Unified). Just click the health check link on the right hand side. I have attached a screen shot below to show what the output of a healthy array looks like.

So what are you waiting for? Get out there and enjoy these new enhancements. Remember, you don’t have to wait for EMC to upgrade your array, you can do it yourself using USM.

VNX iSCSI and TCP Delayed Acknowledgement

By Sean | December 17, 2012 - 9:21 AM |December 17, 2012 DellEMC, VMware, VNX

5 Comments

I recently sat in on an internal VNX (and CLARiiON) performance crash course that was put together to help our new hires get up to speed. Once of the things that stuck out to me was the subject of iSCSI and how it works with host TCP delayed acknowledgement (Delayed ACK).

Background Information

So what is delayed ACK? As part of TCP, for every packet that is sent to a destination server, that server must send some sort of acknowledgement back to the source server. This way the source server knows the information was successfully transmitted. This adds a good amount of overhead, so in an effort to improve performance, TCP Delayed Acknowledgement (RFC 1122) was created which allows a destination server to respond to every 2nd packet instead. This has become so popular, that support for delayed ACK is enabled by default in many popular client operating systems including Microsoft Windows and VMware ESX/i.

The problem with this is that many storage arrays do not support delayed ACK for one reason or another (usualy has to do with chipset drivers). What happens in this case is that the array will send a packet, it will then wait for an acknowledgement before sending a 2nd packet. Meanwhile, the host is waiting for a 2nd packet before sending an acknowledgement. This standoff between the array and the host will last until the acknowledgement timeout (usually around 200ms) before continuing on. This wreaks havoc on performance if every packet has to wait 200 milliseconds before sending another. So if you’ve setup iSCSI and you are immediately seeing a performance issue, check your hosts to see if Delayed ACK is enabled, and turn it off to see if performance improves.

Disabling Delayed ACK in Microsoft Windows

In Microsoft Windows operating systems, you can simply set the TcpAckFrequency registry value to 1. More information can be found Microsoft kb 328890. On a side note, I found that if the registry value is missing, you can create it in the path specified in the kb and reboot the host.

Disabling Delayed ACK in VMware ESX and ESXi

VMware has created KB 1002598 to address this as well. This adjustment is made per adapter instance and you can change this setting on a discovery address, a target, or (in my case) globally. Once you’ve made your change, reboot the host and enjoy the performance boost.

I hope you’ve found this information useful. It may not solve your iSCSI performance problem, but it is a good place to start.

A few initial thoughts about VMware vSphere 5.1

By Sean | September 16, 2012 - 2:54 PM |September 17, 2012 DellEMC, Reviews, VMware

The upgrade process

Previous vCenter server upgrades simply consisted of just upgrading the server component (and the upgrade and away you go. Now VMware has introduced two new components to the mix: vCenter Inventory Service & vCenter Single sign-on. To take a lot of the guess work out of this, VMware provided the “simple installer” to the mix. This runs through each of the setup applications in the proper order and makes the process very simple and I applaud VMware for this. A couple things to note:

As with all vSphere upgrades, make sure you upgrade vCenter server BEFORE you upgrade any hosts.
If you plan on using LDAP authentication, make sure that you are logged in with an LDAP account when you install the single sign-on service.
If an upgrade fails during the “simple install”, you will need to continue installation by hand.
You won’t need to reboot the vCenter Server machine after the upgrade, the upgrade process will restart all the required services for you.
After upgrading the hosts, they did not automatically reconnect to vCenter, and you have to click reconnect on them.

Since I run my vCenter server with an internal database and on a virtual machine, I was able to simply snapshot the VM in the event that something went wrong. Luckily nothing did and both my upgrades went smooth.

vCenter Server 5.1

One of the things I noticed right away was the startup time of vCenter server from boot up. It is at least 4x longer than vCenter server 5.0 and earlier. A simple look at task manager during the start up process shows that the java process is eating up most of my CPU, so my guess is this delayed startup is due to the revamped web interface being used. Needless to say, it threw me for a loop the first time and I thought my upgrade had been borked. So just keep that in mind that it may take 20 minutes or so for vCenter to come up and be stable.

Now when using the standard vSphere client to connect, you will notice things are very similar to older versions. This is be expected as I believe VMware is trying to make a shift to everyone using the Web Interface (more on this later). Do keep in mind that the regular vCenter client does not give you access to all of the new features. One of the big ones to note, is that the new enhanced vMotion can only be found in the web client (again more on this later).

vCenter Web Client

It’s clear that VMware has spent a lot of time improving upon the web interface and it shows. That’s good news for Mac users as they now have a fully functioning method for administering the environment.

Right away, you’ll notice a new icon that is not present in the standard vCenter client. VMware has included vCenter Orchestrator right into the mix and I hope this is a sign of more plugins to be installed. With that being said, I was disappointed to see that the update manager was not present in the web client, so you must still use the classic vCenter client to perform those updates.

One of the things I noticed (and was pointed out by Frank Denneman) was that the new Enhanced vMotion (the one that lets you change both hosts and datastores while the guest is powered on) can ONLY be found in the web client. I’m not sure this was the best play by VMware, but my guess is that VMware’s long term plan is to make the web client the primary interface for management.

Final Thoughts

As with every upgrade, I’m always amazed by the refinement that is introduced. More and more drawn out processes are being simplified and it seems like VMware really cares about the advanced users as well as those just getting started. As with any major upgrade, first adopters have some challenges and I wanted to list out a few things that you should be aware of before you upgrade:

As with all my blog posts, I like to hear from the readers. If you had a good or a bad upgrade experience, let me know in the comments.

Introducing EMC’s VNX Storage Analytics Suite (and Early Access Program)

By Sean | August 28, 2012 - 9:00 AM |August 28, 2012 DellEMC, VMware, VMworld, VNX

1 Comment

With VMworld in full swing, today EMC finalized the announcement of the new VNX Storage Analytics Suite (and you can try it out on the show floor). You may remember that this was demoed first at EMC World. As development was nearing completion, you may remember my call for beta testers (I hope you got in and got a chance to try out the technology). Now we can start getting as excited as the software will be available for general consumption in Q4 of 2012.

The VNX Storage Analytics Suite will offer you a an extensive platform to proactively identify bottlenecks, balance workloads and pinpoint the root cause of potential problems around health and performance. The software is available as a stand alone product as well as integration for VMware’s vCOps Enterprise platform. Pairing the two together allows you to have a full end-to-end visibility of your entire infrastructure from virtual machine to lun and every point in between.

For those of you who are already familiar with vCOps, you will be presented with same interface just with extra options for looking at metrics. The charts and display is the same. Green is still good and Red is still bad. This allows you to quickly visualize the health of your arrays (both block and file) using a simple “Performance-at-a-glance” tab. Armed with this knowledge, you’ll be able to attach the problem directly instead of waiting for someone to tell you there is a slowdown in the environment.

Besides general health information, you can dig down into the specifics of utilization on the array. In the picture to the right, you’ll be able to see the results (both historically and in real time) of enabling fast cache on your array and what it did to the overall health of your environment. By focusing on the performance of the array and the pool disks, Fast Cache was able to increase the throughput of the SP while decreasing the strain on the spinning disk storage.

By this point you might be saying “Sean, this is so cool, I can’t wait until Q4! How can I get my hands on this now?”. Well for those of you who did not participate in the beta (and those of you who did and want to see the results of your feedback) there is good news. EMC is going to be doing an Early Access program, it won’t cost you a penny, and I’ve got all the details right here! The bad news is that it’s limited to 100 customers and it’s filling up fast.

If you may remember during the beta program I was asking people to email me if they wanted in. This time I have no control over the process and you must go through your TC. Before I tell you just how to get in, I need to make sure you have what is required in your environment. You will need to meet the following prerequisites:

vCenter Operations Manager 5.0.1 Enterprise or higher (vApp distribution)
EMC VNX series system (Block: R31 OE version 05.31.000.5.720 and higher and VNX OE R32 version 05.32.000.5.006 and higher; File: VNX OE 7.0.35 and higher).

Seems simple enough, right? Now on how to get in, as I said before you will need to contact your TC or Account Representative. Have him or her place a Sales Evaluation order in Direct Express using Model # SASCONN-T90 and complete an RPQ. Again, this is a no cost evaluation period and if you are selected, you will receive further instructions. At the end of the preview period (which I believe is somewhere in Q4), you will be given the opportunity to purchase a license or uninstall the software.

If you are attending the show, please let me know your thoughts about the software in the comments section below.

Configuring VASA for use with a VNX

By Sean | August 5, 2012 - 4:19 PM |August 7, 2012 DellEMC, VMware, VNX

11 Comments

When VMware introduced vSphere 5 to the world, one of the enhancements was a new API for storage arrays that provides vSphere with information as to the configuration and performance of your storage array. For more information on VASA, please see this article from The Virtualization Practice. VASA on a VNX (and other EMC arrays) historically used to be configured using an SMI-S provider. This older configuration method has been covered very well by EMC vSpecialist Craig Stewart and can be found here.

Starting with VNX OE for FILE 7.1 and VNX OE for BLOCK 05.32, the VNX now has native VASA support. This eliminates the need for the SMI-S provider and allows you to point vSphere directly to the control station and SP. It really is a 1-step implementation and I will show you below. And there is only 1 caviot to this, and it is VASA for the BLOCK and FILE are done separately. if you are using, FC, FCoE, or iSCSI connections, you will want to use the BLOCK example, and if you are using NFS, you will want to use the FILE example.

You will want to start in vSphere by going to Home > Administration > Storage Providers. From there you would click on “add…” to configure your connection.

VNX OE for FILE 7.1 VASA configuration example

You will start by naming this connection. I chose VNX FILE to make it easy to distinguish between block and file connections. You will then use the URL as follows: https://<ip.or.dns.of.control.station>:5989/vasa/services/vasaService. The username/password would be one local to the control station (such as nasadmin or root). The global accounts from the storage domain will not work here. When it’s all said and done you should have something like the photo below:

You will probably be prompted to verify the SHA fingerprint, so just click yes and soon you’ll see your new provider listed with the following information:

VNX OE for BLOCK 05.32 configuration example

Just like the VNX OE for FILE example, you will start off by using a name. This time the URL will be pointing to the SP. The url will be as follows: https://<ip.or.dns.of.SP>/vasa/services/vasaService. Please note the lack of a port specification as by default https uses port 443. For the password you will want to use a storage domain account (such as sysadmin). If you configured it correctly, it should look something like this:

Since I have a very basic configured array in the lab, I see provider information like this:

After you have successfully configured your providers, you can go and setup your storage profiles. Go to Home > Management > VM Storage Profiles and add a new profile. From there you can select from a multitude of options to pick the one that best matches the lun you are using for storage.

It really is that simple! For more information on VASA on the VNX, read the Virtualization for EMC® VNX Release Notes (EMC Support credentials required).

Got a VNX and vCOps? We need your help!

By Sean | July 26, 2012 - 9:30 PM |August 3, 2012 DellEMC, VMware, VNX

1 Comment

If you remember during EMC World 2012, the USD team showed off a preview of our new storage analytics package that would come as either a stand alone application or as a plugin for VMware vCOps. As we move ever closer to a final product, we have entered the private beta phase. This is where we need your help.

USD Engineering will be running a private beta between August 8th and August 29th and the test plan should take about 2 days to complete. What you need is the following (UPDATED):

A VNX running recent code. Both File 7.0 & 7.1 and Block 5.31 & 5.32 should be acceptable (this can be production or non production)
vCOps running the Enterprise or Enterprise+ (this must be a non production / test instance)
The name of your EMC contact (preferably a TC)
Please also indicate if you are a customer or a partner

You may need to sign an NDA, I’m not 100% on that but the team running the beta will be able to confirm. Space is limited and we only have a few slots left so there is no guarantee that you will be accepted into the program. If you are interested, please send an email to “sean.dot.thulin.at.emc.dot.com” with your contact information and what you have available as equipment (make / model of VNX and software version installed) and I will forward it on to the team. Act fast as we are almost ready to start!

I Tech Therefore I Tool Around – Episode 02 – The Home Lab

By Sean | December 27, 2011 - 10:00 AM |December 26, 2011 DellEMC, Social, VMware, VNX

6 Comments

So our first episode of the “I Tech Therefore I Tool Around” was a smashing success! Matthew Brender and I had a great time recording and we learned a lot about the process that goes into making a podcast. The feedback from our listeners was great as well.

With that experience and knowledge in hand, we recorded our second podcast. In this episode, we talk about the why and the how of building a home lab. I touched upon my home lab during my blog post about the VCP4 Our special guest, Luigi Danakos, talks about his need for a home lab and how he acquired one without spending a dime. We also touch on several blog posts that inspired our builds. These are all great resources for building a home lab, so check them out:

A slight apology for Luigi as he was still getting over an illness and can be heard coughing every now and then during the recording (I cut out as much as I could). Be sure to check out Matt’s post on this experience.

Click HERE to subscribe to the podcast on iTunes!

Or click HERE (or on the icon) to download the MP3 directly!

New version of VMware View Client for iPad released!

By Sean | October 24, 2011 - 9:25 AM |October 24, 2011 VMware

Category Archives: Virtualization