I Tech Therefore I Tool Around – Episode 04 – EMC World 2012

EMCWorldIts been a while since Matt and I teamed up to do another installment of the I Tech Therefore I Tool Around Podcast.  With EMC World right around the corner, we decided to preview the upcoming event!  Being located near the EMC Headquarters has it’s benefits, and one of those is that many people were in town working on projects for the show.  We invited two vSpecialists Jase McCarty and Dave Robertson to join us in talking about the show.


In this podcast we talk about several different subjects revolving around EMC World.  We started off by talking about the vLabs, what went into them last year, and what we can look forward to this year.  Jase is going to be head of operations this year at the vLabs and will be the man to make sure things get done.  Dave has been up here working on a VMAX VSA that will be used as vLab demo this year.


Next we moved on to Project R.A.M.P.  Matt takes us through the first big public result of his new role at EMC.  For those of you who don’t know, Matt has transferred from Engineering to the Social Media team here at EMC.  As of this blog post, Project R.A.M.P has gone live on the EMC Community Network but there will be mission for those attending EMC World.


New this year are BUZZ sessions.  These short 20 – 30 minute gatherings are modeled after the TED Talks and will encompass a variety of subjects.  Matt, myself, and several of the support forum legends will be having a BUZZ session on Wednesday, May 23rd @ 10:30 AM PST.  Be sure to join us both in person and online (more on that later).  In addition to the buzz talk, you will find myself and several other ECN members at the EMC eServices / Social Media booth and around the show conducting interviews and reports.  And be sure to come check out the Bloggers Lounge and catch up with other bloggers.


While at this years show, be sure to visit the EMC Proven Professional area where they are offering 50% off the price of exams.  The show floor will always have some great swag to give away, so be sure to bring an extra bag to fill up.  Closing out the show this year will be another live episode of Chad’s World as well as a special concert by Maroon 5!!!!!!!  I look forward to seeing as many people there again this year.


Click HERE to subscribe to the podcast on iTunes!

Or click HERE (or on the icon) to download the MP3 directly!

I Tech Therefore I Tool Around – Episode 03 – Community

Community pinned on noticeboardA new year, a new installment of #ITTITA!  This time we decided to take a new approach to the podcast.  Instead of picking a technical topic and doing a deep dive, we are going to review the past year.  One of the biggest influences of our year was this theme of “Community” it was everywhere and played a big part in our lives.


Community has been a big factor at some of the different conventions I’ve been to this year.  At EMC World we held a virtual #nerdherd and invited the community to interact with us at the convention as well as online via a live video and chat feed.  At the New England VMware User Group meeting a few weeks ago, community was also present with the great conversations as well as user presentations.


2012 is also going to be a great year of community.  We started the year by launching the “Ask the Expert” forum on the EMC Community Network.  In the first event of the year on there, Matt partnered up with Henri to tackle the tough questions about VNXe VMware integration.  There was some excellent conversation as well as a storage deep-dive and we had a record number of views on this single thread.


All in all I think that 2011 was a great year for me.  I launched my blog, I linked up with some great and knowledgeable people, and I even started a podcastwith one of my best friends.  2012 looks to be an even greater year!


Click HEREto subscribe to the podcast on iTunes!

Or click HERE (or on the icon) to download the MP3 directly!

2 Years at EMC

EMCToday marks my 2 year anniversary at EMC.  Back in January of 2010, I literally graduated from college on a Friday and started at EMC on a Monday.  I was enrolled in the Global Services Associate program (GSAP) which started off with a 10 week intensive training program known as “Bootcamp”.  During this great opportunity I learned a lot about all the different products and services EMC has to offer.

My first 3 weeks were dedicated to passing my first certification known as EMC Technology Foundations (Now replaced by Information Storage Management).  This was a tough challenge as I had no experience with enterprise storage being fresh out of college.  After this I began my deeper training in CLARiiON, SAN, and networking protocols.  This program was a great experience and gave me a good foundation for my job in the NAS support lab.

It was decided quickly that my focus within the NAS support lab would be with CIFS, Active Directory, and anti virus.  With roughly 90% of our install base using CIFS, this was going to be a an important part of the organization.  I quickly excelled in this area and have become a subject matter expert.

At EMC World 2011 in Las Vegas, I passed my NAS Specialist certification and then shortly after that I earned my VMware Certified Professional 4 during the summer.  I’ve been out to Utah to train our new employees and participate on the EMC Community Network as well.

So far these first two years have been great for me.  I’ve met a lot of great people and have participated in some great events.  I only see good things to come from the next 2 years.

Introducing EMC Ask The Expert

AskTheExpertLogoA few months ago Mark Browne approached Matt Brender and I with a new idea for the EMC Community Network.  Mark is very big on ECN and if you’ve visited the site, you’ve probably interacted with things Mark has had a hand in.  So Mark pitched us the idea of starting an “Ask the Expert” section on the support forums.  In this space, we would gather a couple of subject matter experts to answer questions on a related topic for about two weeks at a time and maybe follow up with a video recap.


Matt and I both thought this was a great idea and over the next few months we helped Mark flesh out the idea in preparation to present to the approving management structure.  With the help of our friend Michael Chelotti, we recorded a teaser video.  This video will be very similar to a video recap idea where we will talk about some of the topics during the discussion.  You can watch our video below:


I’m proud to report that this idea was well received and Ask the Expert is a go.  Starting today, we launch our first event!   Matt will be joined by Henri Hamalianen and they will talk about configuring and troubleshooting the VNXe front end connections with VMware.  These two weeks are open for anyone to ask Matt and Henri about using their VNXe in a VMware environment this post on ECN.  I urge everyone to checkout the discussion and get involved.  Keep an eye on the schedule for a discussion on the VNX that may be hosted by yours truly.

I Tech Therefore I Tool Around – Episode 02 – The Home Lab

DSC_0022So our first episode of the “I Tech Therefore I Tool Around” was a smashing success!  Matthew Brender and I had a great time recording and we learned a lot about the process that goes into making a podcast.  The feedback from our listeners was great as well.


With that experience and knowledge in hand, we recorded our second podcast.  In this episode, we talk about the why and the how of building a home lab.  I touched upon my home lab during my blog post about the VCP4  Our special guest, Luigi Danakos, talks about his need for a home lab and how he acquired one without spending a dime.  We also touch on several blog posts that inspired our builds.  These are all great resources for building a home lab, so check them out:


A slight apology for Luigi as he was still getting over an illness and can be heard coughing every now and then during the recording (I cut out as much as I could).  Be sure to check out Matt’s post on this experience.


Click HERE to subscribe to the podcast on iTunes!

Or click HERE (or on the icon) to download the MP3 directly!

I Tech Therefore I Tool Around – The Podcast

I think the title says it all. If you’ve been paying attention to my twitter feed, it’s no secret that Matt Brender and I have been working on a Podcast. We have always had great banter between the two of us and now we have the chance to share it with the rest of you. In this first episode, we cover the subject of certifications. We debate how worth while it is to get certifications in todays job market and how a resume looks to perspective employers.

This is our first podcast, so the editing might be a little rough, but the content is pure gold.  I had a great time debating with Matt and i think it showes in the content.  I learned a lot about the creative process and planning that goes into a podcast and i hope to put it to good use in future episodes.  Let me know what you think about it in the comments below and be sure to tune into our next podcast where Matt and I talk about home labs.

Matt has also written a blog post on this which can be read here.

Click HERE to subscribe to the podcast on iTunes!

I Tech Therefore I Tool Around – Episode 01

Reflections on the Utah Call Center

So my two week stay in Utah has come to a close and it has been a great experience.  Our new recruits are well on their way and they are understanding the inner workings of the VNX a lot faster than I did when I started at EMC.   In the short time I was there I watched them grow from handling simple dial home cases to complex issues and high severity situations.  They work well with each other as well as with customers and I sensed a great deal of comradery amongst the group.  While most of them did work together at a previous employer, they worked well with the other people who were new to them.  There are some clear leaders in the group and you could tell that the other coworkers would gravitate towards them when they needed assistance.


The interaction with the culture of Utah allowed me a greater understanding of how the Mormon religion effects the daily lives of the citizens.  It’s not all that we see on TV shows like “Big Love” and “Sister Wives”.  In fact, polygamy goes against the church’s teachings.  Most of the my new coworkers are in fact Mormon.  Some are more religious than others but they aren’t there to preach to me about their faith.  Some have done their missionary work already and have settled down and started a family, while other aren’t as active with the church.


My stay here allowed me a great deal of freedom to travel and see the sights.  I was able to see what was left of the 2002 Olympic games as well as many geological landmarks and formations.  Coming from the east coast, we don’t get mountains like they do out here and I took every chance I got to enjoy the majestic beauty of them.  If you haven’t already, please check out my photos from my trip on my google+ page.


All in all this was a great trip.  I understand that several of my colleagues will be traveling up to visit Hopkinton, MA soon and I hope to be able to offer them the same experience they offered me here.  I think that great things will come from this new group in Utah and I hope to get out here again soon.

EMC Support: The Next Generation

RemoteHelpIts hard to hide the fact that EMC’s sales are up.  There have been announcements about record profits, growth, and installations all around.  With this increased boom of installed systems, EMC is also increasing it’s support presence as well.  A few months back EMC announced it had broken ground on a new 7 million dollar support facility in Utah.  This center will add at least 500 new US based support engineers to help.


You may be asking what this means to you?  Well in my department (Unified Support) we have added over 60+ new members to our staff.  With the later time zone we can offer extended US based support to our west coast customers instead of doing a hand over at 3 PM Pacific.  This also means there will be more North American based personnel handling cases.


Why am I telling you about this? Well I am happy to announce that I will be traveling to this new center to help mentor the new hires.  I will be able to instill upon them my knowledge of Celerra and VNX.  I will be out there for the first two weeks of November, so if any of my readers are in the area and would like to get together for food or drink, leave me a comment or a shout out on twitter.

Understanding the EMC VNX/Celerra AntiVirus Agent (CAVA): Part 2 – Common Errors

This is part 2 of my CAVA blog post series. In this post, I will go through common error messages you could see in the output of server_viruschk. For those of you haven’t already, please check out part 1 where I go line by line through the output of the server_viruscheck command.


Most of these errors have to do with the account used for CAVA. This account is set as the “Log on as” option for EMC Cava in the “services” section of windows.




server_2 :
10 threads started.
1 Checker IP Address(es):        OFFLINE at Sat Aug 20 20:28:33 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version: , ntStatus: OBJECT_NAME_NOT_FOUND
                     AV Engine:
                     Server Name: cava.thulin.local
                     No signature date

Description: ntStatus: OBJECT_NAME_NOT_FOUND means that the cava service is not running on the server.

Solution: Start the EMC CAVA service under the services menu on the AV server.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 5 at Sat Aug 20 21:00:10 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH means that when cava when to connect to the “check$” folder on the cifs server, it ran into an error. In this case, ERROR_AUTH 5 means that the account does not have the viruschecking privilege.

Resolution: Check to make sure that the EMC CAVA process is running under the cava network user and not the Local System account. If this is correct, verify that you gave the CAVA network account the Viruschecking Privilege in the MMC snap in.



server_2 :
10 threads started.
1 Checker IP Address(es):        AV_NOT_FOUND at Sat Aug 20 20:29:59 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Unknown third party antivirus software
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: AV_NOT_FOUND means that CAVA cannot find a running AV process. By default, cava uses a privilege called “Debug Program Rights” to search for the following applications running in memory: SpntSvc.exe, rtvscan.exe, Mcshield.exe, InoRT.exe, SWEEPSRV.SYS, SavService.exe, NTRtScan.exe, and kavfs.exe

Solution: First check to make sure your antivirus software is installed and running. If this is true, then make sure the CAVA account has the Debug Program Rights. By default, this privilege is granted to all local administrators, so add the cava account to the local administrators folder.



server_2 :
10 threads started.
1 Checker IP Address(es):        OFFLINE at Sun Aug 21 17:08:28 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version: , ntStatus: INVALID_PARAMETER
                     AV Engine:
                     Server Name: cava.thulin.local
                     No signature date

Description: ntStatus is throwing an error trying to connect from the Cifs server to the Cava server. This error is caused when the CIFS server specified for CAVA is not joined to AD.

Resolution: Join the cifs server to AD and restart CAVA.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 64 at Sun Aug 21 18:16:05 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH 64 is because there is a kerberos skew error.

Resolution: Make sure the time on the cava server is within 5 minutes of the data mover.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 86 at Sun Aug 21 17:25:31 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Problem: ERROR_AUTH 86 is caused when someone changes the password of the CAVA user in AD, but the cava software is using the old password.

Resolution: Update the password used for the cava account on each cava server. If you attempt to restart cava without updating, cava will fail to start with a logon failure error.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 1265 at Sun Aug 21 16:04:33 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH 1265 is caused when the cava user account has expired in AD. You can verify this if you attempt to login to a remote desktop with the cava user’s credentials.

Resolution: Have a domain admin reset the CAVA account and change it to never expire to keep this problem from returning.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 1326 at Sun Aug 21 17:49:37 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH 1326 occurs when the cava user’s password has expired in AD.

Resolution: Change the cava account password and have a domain admin set it to never expire.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 1331 at Sun Aug 21 17:09:45 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH 1331 is when the cava account object is disabled or logon hours have been put in place to deny logon.

Resolution: Have a domain admin enable the cava account object in AD and confirm that the cava account can logon at all hours of the day.



server_2 :
10 threads started.
1 Checker IP Address(es):        ERROR_AUTH 1909 at Sun Aug 21 17:57:17 2011 (GMT-00:00)
                     MS-RPC over SMB, CAVA version:, ntStatus: SUCCESS
                     AV Engine: Symantec AV
                     Server Name: cava.thulin.local
                     Last time signature updated: Tue May 17 05:55:23 2011 (GMT-00:00)

Description: ERROR_AUTH 1909 occurs when the cava user account has been locked out due to too many invalid logon attempts.

Resolution: Have an AD admin reset the lockout status on the cava network user.


This should cover most of the common errors you will find when cava is running. You may have to check the server logs on cava to see them in the event that cava is turned off. If you have experienced a problem and my resolution does not fix it, please let me know and also open a case with EMC Celerra support.
On a side note, I want to also recognize Daniel Morris for his blog posts on CAVA. I urge you to read the following links to get a good understanding as well.




Configuring LDAP Authentication for Unisphere on the VNX

Whether you are configuring security for corporate compliance, or you want a central repository to manage user access, LDAP integration is becoming a major part of corporate infrastructure. Many of you may not realize this, but the VNX (as well as the older Clariion and Celerra) support LDAP integration, and after reading this blog post you will to. During this post I will cover the different steps (with pictures) required to set up LDAP authentication for VNX for FILE, BLOCK, and Unified.


*UPDATE*  With the release of FILE OE 7.1 and BLOCK OE 5.32, All LDAP settings are now done in the Storage Domain section of Unisphere.  Just follow the directions here to setup LDAP.


To start this process we will need a few things:

  1. The IPs of two domain controllers
  2. The “distinguished name” and password of a service account that can do an LDAP lookup
  3. The name of an active directory group you want to give admin access to (no spaces pleas)
  4. An existing administrator account on the VNX (and the root password for FILE)


Before we begin, you may want to login to the control station CLI as root and run the following command: “/nas/sbin/cst_setup –reset”. This command will regenerate the control station lockbox fingerprint and is usually required on systems where you may have changed the IP or name of the control station. I find it’s best to get this out of the way early instead of proceeding with configuration and finding it needs to be done later since this does not change any settings outside of the scope of this tutorial. More information on this can be found in Primus EMC260883.


Configuring LDAP on VNX for FILE


To start, we will need to login with an administrator account such as nasadmin/systadmin. You will start by clicking on the “settings” tab. On the right hand side you will see link to “Manage File LDAP Domain”, click it.

This section has several entries and is where we configure all the domain information. I have broken this down line by line as well as included a picture.


  1. Domain Name:
    • In this area you will put in the domain name. For this example, I used my domain “thulin.local”
  2.  Primary:
    • This is where you put in the IP address of the first domain controller
  3. Backup:
    •  This is where you put in the IP address of the second domain controller
  4. SSL Enabled:
    • Are you using SSL? If so, click the box. For this example I am not because I don’t have a certificate authority setup in the lab
  5. Port:
    • 389 for LDAP and 636 if your using LDAPS
  6. Directory Service Type:
    • Here you get 3 options (default, custom, and other). Default takes most of the guess work out, but will only work if the service account and all the users and groups exist in the “users” container. The custom option allows you to specify the exact container for the service accounts and the user and group search path. Other is used for non active directory setups (such as OpenLDAP servers). For this example we are using the custom option
  7. User Id Attribute:
    • This is the attribute that represents a user in LDAP, in 99% of Active Directory environments it is “samAccountName” and we will leave it as that here
  8. Distinguished Name:
    • This is where you put the distinguished name of the service account. For this example I just used the administrator account
  9. Account Password:
    • If this needs explaining then I have a nice etch-a-sketch you should be using instead of a VNX.
  10. User Search Path:
    • This is where you specify the path to search for users who will be logging in. If the user is not inside this path, they will not be granted access. I like to search the whole domain because a user cannot exist in more than one spot, and authentication won’t be effected by moving a user inside active directory
  11. User Name Attribute:
    • This is the attribute to search by, we will use “cn” (aka Common Name)
  12. Group Search Path:
    • This is just like above, but for groups instead. The same restrictions apply as well
  13. Group Name Attribute:
    • Again we want to search by the common name
  14. Group Class:
    • You want to search for the “group” class
  15. Group Member:
    • We are searching for a “member” of a group


Once all the information has been populated, hit apply to save it (if you run into an error here, see the statement I made in paragraph 2 and start over). Once this is done we will need to test things, so hit the test button. If everything worked correctly it will say “Test Domain Settings. OK”. If you get “Bind Failed” error, either your IP, Distinguished Name, or password is incorrect. If you get a user or group error, check the search path and try again.


Now that we have configured our authentication protocol, we need to assign a privilege to an AD group. This is done in the in the user management area, so go back to the settings tab, then click on security, then click on user management, and finally “User Customization for File”. This area will present you with 3 tabs: Users, Groups, and Roles. Click on groups and then click create at the bottom. You will now be presented with a screen to make a new group and map it to LDAP.


  1. Group Name:
    • This is a local name for the group. You can call it whatever you want because it ONLY exists on the VNX FILE control station. I chose the name LDAP_Admins
  2. GID:
    • This is where you can specify a GID or just have the system auto select one. I use the default of auto select
  3. Role:
    • This is where you give permissions to the group based on the role. Any user in this group will be given this role/permission level by default. For this example, I chose to give the users the Administrator role.
  4. Group Type:
    1. This is where you would select “LDAP group mapped” and put in the name of the group (in this case serviceAdmins) and the domain name (thulin.local). The group name can’t have any spaces but does support underscores.


At this point all the work on the VNX FILE side is done and it’s time to start on the BLOCK side.


Configuring LDAP on VNX for BLOCK


Setting up LDAP for Block is very similar to the way it was done on the Clariions. Just like with the File side, you will need the same 4 bits of information. To begin, click on the home button in the upper left, then click on the domain tab, and finally click on “Manage LDAP Domain for Block”. This will bring up a window where we can start configuring our LDAP settings. The block side requires you to setup individual domain controllers, and set all the settings on that one server, so click on the “add” button and we’ll get started. You will see several areas to input information and I will go through them:


  1. IP Address
    • This is where you put in the IP of the domain controller
  2. Port
    • 389 for LDAP, 636 for LDAPS
  3. Server Type
    • There are two options: LDAP Server and Active Directory. Make sure to choose “Active Directory” if you’re using an AD environment (most of you will be doing this)
  4. Protocol
    • LDAP or LDAPS
  5. BindDN
    • This is where you put in the Distinguished Name of the service account just like when setting it up for file.
  6. Bind Password
    • Password for the service account
  7. Confirm Bind Password
    • Make sure it matches
  8. User Search Path
    • Just like with File, this is where you would set the search scope to find your users
  9. Group Search Path
    • Just like with File, This is where you set the search scope to find your groups
  10. Add certificate
    • This is where you would upload a root CA certificate for LDAPS. Make sure it’s in base64 encoding


After you have put in all this information, click on the “Role Mapping” tab so we can map an AD group. Once in there you will want to select “Group” from the first pull down. Put in the name of the AD group (in this example I used “ServiceAdmins”), then select the Role from the second pull down (in this case I selected Administrator), and finally click “Add” to add the mapping. Once you have all your mappings, click ok and wait for the confirmation message. Then you want to do this all over again for the second domain controller. Once you have this all set, click “Synchronize”. And that is it!


Configuring LDAP on VNX for UNIFIED


Configuring LDAP for a unified box is no different than the Block and File side.  The only thing you need to remember is that you need to do both, because the authentication will check your LDAP account against both the control station and the service processor.  Both configurations will have to be working correctly to login properly.


Now it is time to test your LDAP login. Logout of Unisphere by clicking the door icon in the upper right. Open Unisphere again and this time put in your AD username and password. Be sure to select “Use LDAP” and click on “Login”. If all your configuration is correct, you will be brought back in to Unisphere. If you get an access denied message, check you username, password, as well as your user and group search paths.


I have included a youtube video published by EMC that shows exactly what I have demonstrated above.

I hope you enjoyed this tutorial and I hope this is the first of many. If you have any questions on what you’ve just seen, or if you have any suggestions for future write-ups, drop a message in the comments below.