Software update, Operating Environment update or firmware update. Those are the most commonly used synonyms to describe the same thing: Software update. On supportbeta.emc.com they call it VNXe Operating Environment update. In Unisphere you can find “Update software” and by clicking that you will see the current System software version. I’ll be talking about software update.
I was planning to post this as the last part of my hands-on experience series. But while I was writing the series I tested the latest software update (2.1.0) on other VNXe and found out there were fixes that would change some of the posts a lot. So I decided go through the software update at this point and write the remaining parts based on the 2.1.0 software version.
Hands-on with VNXe 3300 series [list edited 9/23/11]:
- Initial setup and configuration
- iSCSI and NFS servers
- Upgrade to latest software version: new features and fixed “known issues”
- Storage pools
- Storage provisioning
- VNXe performance
- Wrap up
The first thing is to download the latest software update from support.emc.com. While downloading the file I opened a chat to EMC technical support just to make sure I was good to go with the update on production environment. The answer was no. Because our systems had been up and running for more than 42 days we would have to power down the whole VNXe before we could proceed with the update. Yes, power down dual SP storage system. Seriously? For a while I didn’t know what to think. I was wondering was it because we were running 2.0.3 or because of the 42+ day uptime and the technician confirmed that it was the 42+ days. This is the information that I got last week. It could be different this week, or with other sw version so check with support when planning to do an update.
So we had two VNXe 3300’s on production with over hundred VMs running on them and we had to take all of those VMs down. We also had one VNXe that was waiting to be configured. I decided to do the update on the new VNXe 3300 first and see if it was worth of taking the VMs down and updating the software. Because the new VNXe didn’t have any hosts connected to it yet it was pretty straight forward. In software update 2.1.0 there were big enough fixes and enhancements to Unisphere that I decided to proceed updating the software. I’ll later go through the reasons which lead to this decision.
Before moving to the software update procedure itself the VNXe had to be powered off. The technician gave me 30 step instructions how to do this so here is a short version:
- Stop all I/O to VNXe
- Place both SPs in Service Mode
- Disconnect power from DPE
- Disconnect power from DAE
- Reconnect power on DAE
- Reconnect power on DPE
- Reboot each SP to return to Normal Mode.
When the upload is ready Candidate Version changes from Not Available to 2.1.0. Before it can be installed a Health Check needs to be ran. If the Health Check doesn’t report any errors the software update can be installed by clicking Install Candidate Version.
Issues with the update
I did the update on four VNXe’s and two out of those had some issues with the update. The other VNXe that we had to power down before the update wasn’t that willing to boot the SPs on Service Mode. I followed the instructions that I got from technical support and after stopping the I/O I placed the non-primary SP (A) on Service Mode. SP A didn’t come back up after 20 minutes, not even after an hour. That time I started suspecting that there was something wrong. From Unisphere everything seemed to be normal: In Service System SP B seemed to be primary and in Normal mode and SP A showed an unknown state, which is normal during the reboot. I then found out that the management IP was not responding at all so Unisphere must have shown pages from the browser cache. There was a small indicator about Unisphere not working properly on the lower left corner of the window and also got error “Storage System Unavailable” when trying to execute any commands on VNXe:
The way that the lights were blinking on SP A indicated that it was on Service Mode. Unplugging the SP B management port the management IP started answering and I was able to login to Unisphere. Now Unisphere was showing that SP A was primary and that it was also in Service Mode. SP B was in Normal mode. I then placed SP B in Service Mode and proceeded powering off the VNXe. It seemed that SPs were in some kind of conflict mode. Software update itself went through without problems.
Another issue that I faced was during the software upgrade of VNXe loaded with NL-SAS disks. The update was halfway done when Unisphere gave an error that upgrade has failed:
After restarting the upgrade it went through without any problems.
However, I noticed that the PSL (VNXe OS runs from internal SSD but uses an area called PSL – Persistent Storage Layout – located on the first four disks on DPE) was now located on the 1st, 2nd, 3rd and 5th disk on the DPE. My guess is that the failed software update caused this. I hope this is not going to cause problems later.
Before the update:
After the update:
Fixes and new features
One issue that I criticised in the first part was that none of the support functions were working on version 2.0.3. Those are now fixed in the version 2.1.0. For example clicking How To Videos under Support will open the browser at supportbeta.emc.com and will ask powerlink username and password. My impression was that all the support functions were supposed to be integrated to the Unisphere. Now it is just a link to the supportbeta site. But hey, at least it works now. If it only could use the stored EMC Support Cretentials.
Bigger fix was done to the block size that is used when creating VMware Datastore. In 2.0.3 when VMware Datastore was created and automatically provisoned to ESX server the vmfs partition was created with 1MB block size. This meant that the maximum file size on that vmfs was 256GB. This has been changed on the version 2.1.0 and 8MB block size is used when new VMware Datastore is created.
Nice new feature on System performance monitoring is that now also Network Activity and Volume Activity can be monitored. It used to only show the CPU Activity. Timeframe can be viewed on a scale 48h/24/1h and in addition to those CPU Activity can also be viewed by activity over the last 5 minutes.
Overall the update procedure is easy and takes about an hour if it’s done on newly installed VNXe. If the system has been up and running for over 42 days then scheduled downtime is necessary to complete the update. I would schedule at least four hours of downtime. Powering down and updating can be done in two hours if everything goes without problems. It took me about 3.5 hours to complete the update with one of the VNXe’s. This is something that should be fixed soon. Isn’t the point of a dual SP storage system the ease of serviceability: the other SP can be updated/replaced while the other SP takes care of the I/O. Currently that is not the case.
VNXe is still a fairly new product and I would think that it’s still heavily developed and software updates will be released quite often. To fix issues and also to add some new features. So it’s good to keep an eye on the supportbeta.emc.com downloads section for the new releases. It is also important to contact EMC technical support before installing new updates on VNXe just to check that it’s ok to update or if there are some extra steps that need to be done. Of course I recommend also reading the release notes first.
In the fourth part I will look in to the storage pools and storage provisioning.