Tag Archives: VNXe

Changing MTU on VNXe disconnects datastores from ESXi


While testing the VNXe 3100 (OE 2.1.3.16008) I found a problem when changing the MTU settings for link aggregate. With specific combination of configurations, changing the MTU causes the ESXi (4.1.0, 502767) to loose all iSCSI datastores and even changing the settings back the datastores are still not visible on ESXi. VNXe also can’t provision new datastores to ESXi while this problem is occurring. There are a couple of workarounds for this but no official fix is available to avoid this kind of a situation.

How did I find it?

After the initial configuration I created link aggregate from two ports, set the MTU to 9000 and also created one iSCSI server on SP A with two IP addresses. I then configured ESXi also to use MTU 9000. Datastore creation on VNXe side went through successfully but on the ESXi side I could see an error that the VMFS volume couldn’t be created.

I could see the LUN under iSCSI adapter but manually creating a VMFS datastore also failed. I then realized that I hadn’t configured jumbo frames on the switch and decided to change the ESXi and VNXe MTUs back to 1500. After I changed the VNXe MTU the LUN disappeared from ESXi. Manually changing the datastore access settings from VNXe didn’t help either. I just couldn’t get the ESXi see the LUN anymore. I then tried to provision a new datastore to ESX but got this error:

Ok, so I deleted the datastore and the iSCSI server and then recreated the iSCSI server and provisioned a new datastore for the ESXi without any problems. I had a suspicion that the MTU change caused the problem and tried it again. I changed the link aggregation on VNXe from 1500 to 9000 and after that was done the datastore disappeared from ESXi. Changing MTU back to 1500 didn’t help, the datastore and LUN were not visible on ESX. Also creating a new datastore gave the same error as before. Datastore was created on VNXe but was not accessible from ESXi. Deleting and recreating datastores and iSCSI servers resolved the issue again.

What is the cause of this problem?

So it seemed that the MTU change was causing the problem. I started testing with different scenarios and found out that the problem was the combination of the MTU change and also the iSCSI server having two IP addresses. Here are some scenarios that I tested (sorry about the rough grammar, tried to keep the descriptions short):

Link aggregation MTU 1500 and iSCSI server with two IP addresses. Provisioned storage works on ESXi. Changing VNXe link aggregation MTU to 9000 and ESXi lose connection to datastore. Change VNXe MTU back to 1500 and ESXi still can’t see the datastore. Trying to provision new datastore to ESXi results an error. Removing the other IP address doesn’t resolve the problem.

Ling aggregation MTU 1500 and iSCSI server with two IP addresses. Provisioned storage works on ESXi. Removing the other IP from iSCSI server and changing MTU to 9000. Datastore is still visible and accessible from ESXi side. Changing MTU back to 1500 and datastore is still visible and accessible from ESXi. Datastore provisioning to ESXi is successful. After adding another IP address to iSCSI server ESX loses the connection to datastore. Provisioning new datastore to ESXi results an error. Removing the other IP address also doesn’t resolve the problem.

Ling aggregation MTU 1500 and iSCSI server with one IP address. Provisioned storage works on ESX. Change MTU to 9000. Datastore is still visible and accessible from ESXi side. Changing MTU back to 1500 and datastore is still visible and accessible from ESXi. Datastore provisioning to ESXi is successful. After adding another IP address to iSCSI server ESX loses the connection to datastore. Provisioning new datastore to ESXi results an error. Removing the other IP doesn’t resolve the problem.

Link aggregation MTU 1500 and two iSCSI servers on one SP both configured with one IP. One datastore on both iSCSI servers (there is also an issue getting the datastore on the other iSCSI server provisioned, see my previous post). Adding a second IP for the first iSCSI server and both datastores are still accessible from ESXi. When changing MTU to 9000 ESX loses connection to both datastores. Changing MTU back to 1500 and both datastores are still not visible on ESXi. Also getting the same error as previously when trying to provision new storage.

I also tested different combinations with iSCSI servers on different SPs and if SPA iSCSI server has two IP addresses and SPB iSCSI server has only one IP and the MTU is changed then the datastores on SPB iSCSI server are not affected.

How to fix this?

Currently there is no official fix for this. I have reported the problem to EMC support and demonstrated the issue to EMC support technician and uploaded all the logs, so they are working on trying to find the root cause of this.

Apparently when an iSCSI server has two IP addresses and the MTU is changed the iSCSI server goes to some kind of “lockdown” mode and doesn’t allow any connections to be initiated. Like I already described the VNXe can be returned to operational state by removing all datastores and iSCSI servers and recreating them. Of course this is not an option when there is production data on the datastores.

EMC support technician showed me a quicker and a less radical workaround to get the array back to operational state: Restarting the iSCSI service on the VNXe. CAUTION: Restarting iSCSI service will disconnect all provisioned datastores from hosts. Connection to datastores will be established after the iSCSI service is restarted. But this will cause all running VMs to crash.

The easiest way to restart iSCSI service is enabling the iSNS server from iSCSI server settings, giving it an IP address and applying changes. After the changes are applied iSNS server can be disabled. This will trigger the iSCSI service to restart and all datastores that were disconnected are again visible and usable on ESXi.

Conclusions

After this finding I would suggest not to configure iSCSI serves with two IP addresses. If MTU change can do this much damage what about other changes?

If you have two iSCSI servers with two IP addresses I would advise not to change MTU even if it would be done during a planned service break. If for some reason it is mandatory to do the change, contact EMC support before doing it. If you have arrays affected by this issue I would encourage to contact EMC support before trying to restart the iSCSI service.

Once again I have to give credit to EMC support. They have some great people working there.


Hands-on with VNXe 3100


On Friday before Christmas we ordered VNXe 3100 with 21 600GB SAS drives and it was delivered in less than two weeks. Exactly two weeks after the order was placed we had the first virtual machine running on it.

Last year I made a seven post series about my hands-on experience with VNXe 3300. This is my first VNXe 3100 that I’m working with and also my first VNXe unboxing. With the 3300s I relied on my colleagues to do all the physical installation because I was 5000 miles away from the datacenter. With this one I did everything by myself from unboxing to installing the first VM.

My previous posts are still valid so in this post I’ll try to concentrate on the differences between the 3300 and 3100. Will Huber has really good posts on unboxing and configuring the VNXe 3100. During the installation I also found a couple of problems from the latest OE (2.1.3.16008). I will describe one of the issues in this post and for the bigger thing I will do a separate post.

I will also do a follow up post about the performance differences between 3300 and 3100 when I get all the tests done. I’m also planning to do some tests with FusionIO IOturbine and I will post those results when I get the card and the tests done.

Initial setup

VNXe and the additional DAE came in two pretty heavy boxes. Which box to open first? Well the box that you need to open first tells you that:

So like a kid on Christmas day I opened the boxes and the first thing that I see is this big poster explaining the installation procedure. The rack rails are easy and quick to install. The arrays are quite heavy but managed to lift those on the rack also by myself.

After doing all the cabling it was time to power on the VNXe. Before doing this you need to decide how you are going to do the initial configuration (assigning IP). In my previous post I mentioned that there are two options for doing it using VNXe ConnectionUtility: auto discovery or manual configuration. With the manual configuration the VNXe ConnectionUtility basically creates a text file on the USB stick that will be inserted into the VNXe before the first boot. A faster way is to skip the download and installation of the 57mb package and create the file manually on a USB stick. So get a USB stick, create IW_CONF.txt file on the USB stick and add the following content to it replacing the [abcd] variables with your own:

TYPE=CONFIGURE

PROTO_VERSION=1

FRIENDLYNAME=[VNXENAME]

MGMTADDRESSA1=[IP ADDRESS]

MGMTMASK1=[NETMASK]

GATEWAY=[GATEWAY]

After that just insert the USB into the VNXe and power it on. The whole process of unboxing, cabling and powering on took me about one and a half hours.

While the VNXe was starting up I downloaded the latest Operating Environment (2.1.3.16008) so that I was ready to run the upgrade after the system was up and running. After the first login ‘Unisphere Configuration Wizard’ will show up and you need to go through several steps. I skipped some of those (creating iscsi server, creating storage pool, licensing) and started the upgrade process (see my previous post).

After the upgrade was done I logged back in and saw the prompt about the license. I clicked the “obtain license” button and a new browser window opened and following the instructions I got the license file. I’ve heard complaints about IE and the licensing page not working. The issue might be browser popup blocker. It is also stated on the license page that the popup blocker should be disabled.

After this quick LACP trunk configuration on the switch and ESXi side iSCSI configurations I was ready to provision some storage and do some testing.

Issues that I found out

During the testing I found an issue with the MTU settings when using iSCSI. A problem that will cause datastores to be disconnected from ESXi. Even reverting back to the original MTU settings the datastores can’t be connected to ESXi and new datastores can’t be created. I will describe this in a separate post.

The other issue that I found was more cosmetic. When having two iSCSI servers on the same SP and provisioning the first VMware storage on the second iSCSI server Unisphere will give the following error:

For some reason VNXe can’t initiate the VMFS datastore creation process on ESXi. But LUN is still visible on ESXi and VMFS datastore can be manually created. So it’s not a big issue but still annoying.

Conclusions

There seems to be small improvements to the latest operation environment (2.1.3.16008). Provisioning storage to ESX server feels a bit faster and also VMFS datastore is also created every time if having only one iSCSI server on SP. In the previous OE the VMFS datastore was created about 50% of the time when storage was provisioned to ESXi.

In the previous posts I have mentioned how easy and simple VNXe is to configure and the 3100 is no different from the 3300 from that point of view. Overall VNXe 3100 seems to be a really good product considering the fairly low price of it. A quick look at the performance tests shows quite similar results to the ones that I got from the 3300. I will do a separate post about the performance comparison of these two VNXes.

Although it is good to keep in mind the difference between marketing material and the reality. VNXe 3300 is advertised to have 12GB memory per SP but in reality it has only 256MB read cache and 1GB write cache. 3100 is advertised to have 8GB memory but it has only 128MB read cache and 768MB write cache.


Hidden VNXe performance statistics


The latest Operating Environment upgrades have already brought some improvements to the statistics that are shown through the Unisphere GUI. The first VNXe OE that I worked with was showing only CPU statistics. Then along with update 2.1.0 Network Activity and Volume Activity statistics came available. I was still hoping to get some more statistics. IOps and latency graphs would have been nice additions. So I did some digging and found out that there is actually lots of statistics parameters that VNXe gathers but those are just stored in the database, maybe for support purposes.

Where is the data stored?

When logging in to the VNXe via SSH using service account and listing the content of the folder /EMC/backend/perf_stats you will see that there are several db-files in that folder.

Now when opening the file with notepad it is quite clear what kind of databases those are:

How to read the data?

Now that we know that the data is stored in SQLite database the next thing is to export the data to readable format. To do this SQLite shell is needed. SQLite is really simple to use, just download shell and run a couple of commands.

To open the database, to select the output file and to export all the data can all be done with using only three commands:

Now all the content of the database is exported to stats_basic_summary.txt. Data can now be imported to spreadsheet or to another database.

What data is stored in the databases?

Actually there is a lot of parameters and data in those databases. Here is just few of the parameters.

DART parameters in stats_basic_default.db:

SysClockUnixms
NetBasicBytesIn
NetBasicBytesOut
NetInPackets
NetOutPackets
TCPInPackets
TCPOutPackets
UDPInPackets
UDPOutPackets
StoreReadBytes
StoreWriteBytes
StoreReadRequests
StoreWriteRequests

DART parameters in stats_basic_summary.db:

NetBasicBytesIn
NetBasicBytesOut
NetInPackets
NetOutPackets
TCPInPackets
TCPOutPackets
UDPInPackets
UDPOutPackets
StoreWriteBytes
StoreReadBytes
StoreReadRequests
StoreWriteRequests
KernelBufCacheHits
kernelBufCacheLookups
CifsActiveConnections
CifsTotalConnections
CifsBasicReadBytes
CifsBasicReadOpCount
CifsBasicWriteBytes
CifsBasicWriteOpCount
FsDnlcHits
FsDnlctotal
FsOfCachehits
FsOfCachetotal
NfsActiveConnections
NfsBasicReadBytes
NfsBasicReadOpCount
NfsBasicWriteBytes
NfsBasicWriteOpCount
iSCSIBasicReads
iSCSIReadBytes
iSCSIBasicWrites
iSCSIWriteBytes

FLARE_SP parameters in stats_basic_summary.db:

HardErrorCount
HighWaterMarkFlushOff
IdleFlushOn
LowWaterMarkFlushOff
writeCacheFlushes
writeCacheBlocksFlushed
ReadHitRatio
SPTimestamp
SumOfQueueLengths
arrivalsToNonzeroQueue
SumOfLUNBlkRead
SumOfLUNBlkWrite
SumOfLUNDiskRead
SumOfLUNDIskWrite
SumOfLUNDiskBlkRead
SumOfLUNDiskBlkWrite
SumOfFRUBlkRead
SumOfFRUBlkWrite
SumOfFRUReadCount
SumOfFRUWriteCount

How can that data be used?

I take the StoreReadRequests parameter from stats_basic_default.db as an example. Some of the parameters have descriptions and this is one of those:

Total number of read requests on all DART volumes

Here is the format that the data is in after imported to spreadsheet:

There is a time stamp and also a value for the StoreReadRequests. It seems that the number of read requests that were recorded during the five minute period is added to the old value and then inserted as a new entry to the database. So basically subtracting the the earlier value from the new one we get the total number of read requests for all DART volumes for the specific five minute period of time:

4267021177 – 4266973002 = 48175

Now if we divide that result with 300 (seconds) we get the average number of read requests on all DART volumes per second during the specific five minute period:

48175 / 300 = 160.58

With some spreadsheet magic it is easy to create a nice “requests per second” graph from the data:

How can I be sure that my theory is correct?

Well, NetBasicBytesIn and NetBasicBytesOut parameter values in the stats_basic_default.db are also growing with every time stamp. These are also defined in the database: Total Bytes DART received/sent from all NICs. So I used the same math to do a graph showing network statistics for the past 24 hours. I then compared that graph with the Unisphere’s network activity graph and those were matching.

The graph that I put together using the values from the database and the formula  introduced earlier:

Unisphere network activity graph:

Conclusions

I really hope that EMC will bring more statistics to the GUI or introduce a way to export the data to readable format a bit easier. From what I’ve heard Clint Kitson from EMC has already wrote some scripts for pulling the stats from VNXe but it is not yet published for the customers. Digging into the databases is kind of a quick and dirty way to get more statistics out of the VNXe, but it seems to be working.


VNXe 3300 performance follow up (EFDs and RR settings)


On my previous post about VNXe 3300 performance I introduced results from the performance tests I had done with VNXe 3300. I will use those results as a comparison for the new tests that I ran recently. In this post I will compare the performance difference with different Round Robin policy settings. I also had a chance to test the performance of EFD disks on VNXe.

Round Robin settings

On the previous post all the tests were ran on default RR settings which means that ESX would send 1000 commands through one path before changing the path. I observed that with the default RR settings I was only getting the bandwidth of one link on the four port LACP trunk. I got some feedback from Ken advising to change the default RR IO operation limit setting from 1000 to 1 to get two links worth of bandwidth from VNXe. So I wanted to test what kind of an effect would this change have on performance.

Arnim van Lieshout has a really good post about configuring RR using PowerCLI and I used his examples for configuring the IO operation limit from 1000 to 1. If you are not confident running the full PowerCLI scripts Arnim introduced in his post here is how RR settings for individual device could be changed using GUI and couple of simple PowerCLI commands:

1. Change datastore path selection policy to RR (from vSphere client – select host – configure – storage adapters – iSCSI sw adapter – right click the device and select manage paths – for path selection select Round Robin (VMware) and click change)

2. Open PowerCLI and connect to the server

Connect-VIServer -Server [servername]

3. Retrieve esxcli instance

$esxcli = Get-EsxCli

4. Change device IO Operation Limit to 1 and set Limit Type to Iops. [deviceidentifier] can be found from vSphere client’s iSCSI sw adapter view and is in format of naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

$esxcli.nmp.roundrobin.setconfig($null,”[deviceidentifier]”,1,”iops”,$null)

5. Check that the changes were completed.

$esxcli.nmp.roundrobin.getconfig(“[deviceidentifier]”)

Results 1

For these tests I used same environment and Iometer settings that I described on my Hands-on with VNXe 3300 Part 6: Performance post.

Results 2

For these tests I used the same environment except instead of virtual Win 2003 I used virtual Win 2008 (1vCPU and 4GB memory) and the following Iometer settings (I picked up these settings from VMware Community post Open unofficial storage performance thread):

Max Throughput-100%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 100% Read distribution
  • 5 minute run time

Max Throughput-50%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 50% read/write distribution
  • 5 minute run time

RealLife-60%Rand-65%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 40% sequential / 60% random distribution
  • 35 % read /65% write distribution
  • 5 minute run time

Random-8k-70%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 100% random distribution
  • 30 % read /70% write distribution
  • 5 minute run time
[Updated 11/29/11] After I had published this post Andy Banta gave me a hint on twitter:
You might squeeze more by using a number like 5 to 10, skipping
some of the path change cost.
So I ran couple of more tests changing the IO operation limit between 5-10. With the 28 disk pool there was no big difference when used the values 1 or 5-10. With the EFDs the magic number seemed to be 6 and with that I managed to get 16 MBps and 1100 IOps more out of the disks with specific work loads. I added the new EFD results to the graphs. 


Conclusions

Changing the default RR policy setting from the default 1000 io to 1 io really makes a difference on VNXe. On random workload there is not much difference between these two settings. But with sequential workload the difference is significant. Sequental write IOps and throughput is more than double with certain block sizes when using the 1 io setting. If you have ESXs connected to VNXe with LACP trunk I would recommend changing the RR policy to 1 5-10. Like I already mentioned Arnim has a really good post about configuring RR settings using PowerCLI. Another good post about multipathing is A “Multivendor Post” on using iSCSI with VMware vSphere by Chad Sakac.

Looking at the results it is obvious that EFD disks perform much better than SAS disks. On sequential workload 28 Disk SAS pool’s performance is about the same as 5 disk EFD RG’s. But on random workload EFD’s performance is about two times better than SAS pool’s. There was no other load on the disks while these tests were ran so under additional load I would expect EFD’s performing much better on sequential load as well. Better performance doesn’t come withouth a bigger price tag. EFD disks are still over 20 times more expensive per TB than SAS disks but then again SAS disks are about 3 times more expensive per IO than EFD disks.

Now if only EFDs could be used as cache on VNXe.

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.


VNXe Operating Environment 2.1.1


[Edited 10/12/2011] Apparently EMC has pulled back the VNXe OE MR1 SP1 (2.1.1.14913) and it is not available for download anymore. According to EMC new image and release notes will be available soon.      

EMC has released VNXe operating environment version 2.1.1.14913 (2.1 SP 1) for VNXe 3100 and 3300. And how did I find out about the new update? Well my VNXe told me about it:

Release notes and software are available on supportbeta.emc.com. The first thing that I noticed on the short description on the upgrade package was that VNXe OE has to be 2.1.0.14097 or higher before upgrading to 2.1.1. On the release notes I couldn’t find any mention about this. The only mention about mandatory upgrade is that VNXe should be upgraded to version 2.0.2 or later within 42 days of the initial VNXe installation or otherwise the system has to be powered off before upgrade (KB article emc265195). I also mentioned about this issue on my previous post about VNXe software update. So I contacted the support using chat and quickly got a confirmation that the VNXe has to be on 2.1.0.14097 before upgrading to 2.1.1.

Here is a quick pick of the new features, enhancements and fixes. Full descriptions can be found from the release notes.

New features and enhancements

6+1 RAID5 is now supported on VNXe 3100 with SAS drives and user-defined pools. Automatic configuration will still use 4+1 RAID5 for SAS drives.

EFD drives and 1TB NL SAS drives are now supported on VNXe 3300 DPE and DAE.

There have also been improvements to Unisphere performance.

Fixed

  • Looping problem that might cause SP to boot when network or power cable is disconnected
  • Issues with email alerts
  • Issues with password reset button causing SP to reboot
  • Error with hidden shared folders
  • VMFS datastore creation issues

There is also a long list of known problems and limitations. Couple of those concern VMware integration and are good to keep in mind:

  • VMFS datastore created from VNXe will be VMFS3 an use 8MB blocksize.
  • Manual rescan for storage is required after deleting datastore from standalone ESX server

Hands-on with VNXe 3300 Part 7: Wrap up


When I started writing this series my first plan was to just make four quick posts about my experience with EMC VNXe 3300. After I started going further into detail on the configuration I realized that I couldn’t get everything that I wanted to express to fit in four posts. So I ended up adding three more posts to the series. Still, with this series I’m just touching the surface of VNXe 3300. There are so many functionalities that I didn’t go through or didn’t even mention. That was also my intention with this series. I wanted to write about my experience and look in to those features that I would implement.

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

EMCWORLD_39

Simple, Efficient, Affordable

Those are the words that EMC uses for marketing VNXe and I can mostly agree that those are accurate adjectives for VNXe. In the first part I also wanted to add the adjective “easy” among those words. A user can do the initial setup and VNXe can be operational in less than an hour depending on the software version. Unisphere UI layout is also very user friendly and illustrative. Configuration and updating SW are easy and simple.

EMCWORLD_42

Customers buying VNXe just based on the marketing material might face a big surprise when looking in to the actual memory configuration. Yes, VNXe has 12GB memory per SP, but only 256MB is dedicated for read cache and 1GB dedicated for write cache.

Configuration considerations

Even though it is easy and simple to get VNXe up and running and to start provisioning storage this doesn’t mean that the planning phase can be ignored. User can easily be in a really bad situation and the only way out is to delete all datastores to do the proper reconfigurations. Creating only one iSCSI server and putting all datastores on that one server creates a situation where all the I/O goes through one SP and the other SP is idle. Depending on the ESX iSCSI settings only one network port on VNXe could be utilized even if a four port trunk was configured. Fixing this problem is not as easy as creating it. VNXe doesn’t allow changing the datastore iSCSI server after the datastore is created. To assign a different iSCSI server for a datastore it has to be deleted and recreated. This is, again, one issue that I’m hoping will be fixed.

When using four 1GB ports my suggestion would be to configure NIC aggregation on VNXe as I described in part 2. For the ESX configuration I would suggest reading the detailed comment Ken posted in part 6 about the ESX iSCSI configurations. What comes to the VNXe iSCSI and datastore configurations I ended up creating equal number of datastores for each SP and also dedicating one iSCSI server per datastore to get the most out of the four port trunk.

Issues

The issues that I faced during the configuration were mostly minor usability flaws and some of those were already fixed in the latest software version. The biggest issue that I found was that the VNXe had to be powered off before a software update if it had been running for more than 42 days. I’ve discussed these issues with EMC and hopefully they will be fixed in the future releases.

Conclusions

Despite all the criticism I think that VNXe 3300 is a great product and it will be even better when the few small flaws are fixed. I’m really looking forward to seeing what kind of new features will be introduced in the future software releases. Chad Sakac gave a hint on his blog post about FAST VP support coming in to VNXe at some point. He also mentioned that VAAI (file) and SRM support will be coming out still this year.

I can see some new VNXe 3300 related blog posts in my near future but I think it is time to close this series and keep the new posts separate. If you have any questions about my experience with VNXe or other general questions about it please leave a comment.


Hands-on with VNXe 3300 Part 6: Performance


Now that the VNXe is installed, configured and also some storage has been provisioned to ESXi hosts it is time to look at the performance. Like I mentioned in the first post I had already gathered some test results from CX4-240 using Iometer and I wanted to make similar tests with VNXe so that the results could be comparable.

Hands-on with VNXe 3300 series:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

Test environment CX4-240

  • EMC CX4-240
  • Dell PE M710HD Blade server
  • Two 10G iSCSI NICs with total of four paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O  paths
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Test environment VNXe 3300

  • EMC VNXe 3300
  • Dell PE M710HD Blade server
  • Two 10Gb iSCSI NICs with total of two paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O paths (see Trunk restrictions and Load balancing)
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Iometer Configuration

I used Iometer setup described in VMware’s Recommendations for Aligning VMFS Partitions (page 7) document.

Disk configuration

I had to shorten the explanations on the charts so here are the definitions:

  • CX4 FC 15D
    • 15 15k FC Disk RAID5 Pool on CX4-240 connected with iSCSI
  • CX4 SATA 25D
    • 25 7.2k SATA Disk RAID5 Pool on CX240 connected with iSCSI
  • VNXe 21D 2.0.3
    • 21 15k SAS Disk RAID 5 (3×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.0.3
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300. connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.1.0
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.1.0
  • VNXe RG 2.1.0
    • 7 15k SAS RAID5 (6+1) RG on VNXe connected with iSCSI. VNXe Software version 2.1.0
  • VNXe 28D NFS
    • 28 15k SAS RAID 5 (4×6+1) Pool on VNXe 3300 connected with NFS. VNXe Software version 2.1.0

100Gb thick LUN was created to each pool and RG where 20Gb virtual disk was stored. This 20Gb virtual disk was presented to the virtual Windows server that was used to conduct the test. Partition to this disk was created using diskpart command ’create partition primary align=1024′ and partition was formatted with a 32K allocation size.

Trunk restrictions

Before I go through the results I want to address the limitation with the trunk between VNXe and the 10Gb switch that it is connected to. Even though there are 4Gb (4x1Gb) trunk between the storage and the switch the maximum throughput is only the throughput of one physical port.

While I was running the tests I had SSH connection open to VNXe and I ran netstat -i -c command to see what was going on with the trunk and individual ports. The first screen capture is taken while 8k sequential read test was running. You can see that all the traffic is going through that one port:

The second screen capture is taken while the VNXe was in production and several virtual machines were accessing the disk. In this case the load is balanced randomly between the physical ports:

Load balancing

VNXe 3300 is active/active array but doesn’t support ALUA. This means that LUN can only be accessed through one SP. One iSCSI/NFS server can only have one IP and this IP can only be tied to one port or trunk. Also LUN can only be served by one iSCSI/NFS server. So there will be only one path from the switch to VNXe. Round robin path selection policy can be enabled on ESX side but this will only help to balance the load between the ESX NICs. Even without the trunk round robin can’t be used to balance the load between the four VNXe ports.

Test results

Each Iometer test was ran twice and the results are the average of those two test runs. If the results were not similar enough (i.e. several hundred difference in IOps) then a third test was ran and the results are the average of those three runs.

Same as previous but without NFS results:

Average wait time

Conclusions

The first thing that caught my eye was the difference between VNXe 28 disk pool and 7 disk RG on the random write test. Quote from my last post about the pool structure:

When LUN is created it will be placed on the RAID group (RG) that is the least utilized from a capacity point of view. If the LUN created is larger than the free space on an individual RG the LUN will be then extended across multiple RGs but there is no striping involved.

The tests were ran on 100Gb LUN so it should fit in one RG if the information that I got would be correct. So comparing the pool and the results from one RG random write test it seems that even smaller LUNs are divided across multiple RGs.

Another interesting detail is the difference of the software version 2.0.3 and 2.1.0. Looking at the difference of these results it is obvious that the software version has a big effect on the performance.

NFS storage performance with random write was really bad. But with the 1k sequential read it surprised giving out 30000 iops. Based on these test I would stick with the iSCSI and maybe look at the NFS again after the next software version.

Overall VNXe is performing very well compared to CX. With this configuration the VNXe it is hitting the limits of the one physical port. This could be fixed by adding 10Gb I/O modules. Would be nice run the same test with the 10Gb I/O modules.

We are coming to an end of my hands-on series and I’ll be wrapping up the series in the next post.

[Update 2/17/2012] Updated NFS performance results with VNXe 3100 and OE 2.1.3.16008: VNXe 3100 performance

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.


Hands-on with VNXe 3300 Part 5: Storage provisioning


When provisioning storage on VNXe there are many options depending on the host type: Microsoft Exchange, Shared folders, Generic iSCSI, VMware or Hyper-V . Because this VNXe is only going to serve VMware ESXi hosts I’m going to concentrate on that part. I will go through provisioning storage from VNXe and also using VSI Unified Storage Management Plug-in.

Hands-on with VNXe 3300 series [list updated 9/23/2011]:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

iSCSI datastore

In the last part a storage pool was created. The next step is to create datastores on it and provision those to the hosts. There are two options when provisioning datastores for VMware from VNXe: VMware datastore or generic iSCSI. When using the VMware storage VNXe will automatically configure the iSCSI targets to selected ESX hosts and also create the VMFS datastore. If generic iSCSI is created then all those steps have to be done manually to each ESX. I really recommend using the VMware storage from these two options. VMware storage can be created from Storage – VMware storage.

At this point only iSCSI server was configured so the only option was to create a VMFS datastore.

The next step is to select the iSCSI server and the size for the datastore. When creating the first datastore it really doesn’t matter which iSCSI server is selected. If the iSCSI_A (on SP A) server is selected for the first datastore then iSCSI_B (on SP B) should be selected for the second datastore to balance the load on the SP’s. When selecting the iSCSI server on SP A this means that all the datastore I/O will go through the SP A. If all the datastores are placed on one SP there could be a situation where the whole VNXe performance is impacted because all I/O goes through that SP and the other SP is idle. So it is important to balance the datastores between SPs. VNXe is not doing this automatically so the user has to manually do this when creating datastores. If datastores are distributed between the SPs and the other SP fails all the datastores on the failed SP’s iSCSI server are moved to the other SP. When the failed SP comes back online all the datastores originally located on it are moved back.

There is an option to configure protection for the storage. Because this datastore is only for testing I chose not to configure the protection.

Step 5 is to select the hosts that the datastore will be attached to. Datastore can be connected to a specific iSCSI initiator on ESX server by expanding the host and selecting Datastore access for the specific IQN. If Datastore access is selected from the host level then the VNXe targets are added to all iSCSI initiators on ESX.

After these steps are completed VNXe starts creating the datastore, adding iSCSI targets to the selected ESX hosts and iSCSI initiators on those and finally mounts and creates the VMFS datastore.

 iSCSI datastore issues

After creating the first datastore I noticed from vCenter that there were “Add Internet SCSI send targets” and “Rescan all HBAs” tasks showing on the hosts that I selected to add the datastore to. After watching those tasks looping for 15 minutes and datastore not showing up on the ESXi servers I figured out that there was something wrong with the configuration.

I found out that the ESXi server had datastores connected also from other storage units that use CHAP authentication. On the ESXi iSCSI initiator the CHAP settings were set to “Inherit from parent” and that meant all the new targets would also inherit these CHAP settings. After disabling the inheritance the new datastore was connected and the VMFS datastore was created on it. I haven’t tried to use CHAP authentication with VNXe datastores so I don’t know if those settings are automatically configured to ESX. VNXe already has the ability to manipulate the ESX server configuration so I would imagine that it could also be possible to change the iSCSI target option to “Do not use CHAP” when VMware datastore is created on VNXe without CHAP authentication. Maybe in next the software version?

Another issue I had was that  a VMFS datastore was not always created during the process of creating a VMware datastore. VMware Storage Wizard on VNXe indicated that creating a VMware datastore is completed but actually the “create VMFS datastore” task was never initiated on the ESX. I’ve created over 20 datastores using this method and I would say that in about 50% of those cases the VMFS datastores were also created. Not a big thing, but still an annoying small glitch.

NFS datastore

Creating an NFS datastore is very similar to creating an iSCSI datastore. The only differences are steps 2 and 5 on the VMware Storage Wizard. On step 2 NFS is selected instead of VMFS (iSCSI). On this page there are two “hidden” advanced options: Deduplication and Caching. These options are hidden under the “Show advanced” link – similar to what criticized in iSCSI server configuration. In my opinion these should be shown by default.

On step 3 the same rule applies to selecting the NFS server as with the iSCSI server. The user has to manually balance the datastores between the SPs.

On step 5 datastore access (no access, read-only or read/write) will be chosen for the host or a specific ip address on host.

When all the steps on the wizard are done VNXe creates the datastore and mounts it to the selected host or hosts. I only gave one host access to this new NFS datastore but I could see that VNXe tried to do something NAS related to all the hosts in the Virtual Center connected to VNXe and gave some errors:

VSI

In a nutshell VSI Unified Storage Management (USM) is a plug-in that integrates with VMware vCenter and can be used to provision storage using vCenter UI. There is lots of good documentation on EMC Unified Storage Community and Powerlink so I’m not going to dig any deeper into it. VSI USM can be downloaded from Powerlink – Support – Software Downloads and Licensing – Downloads T-Z – Virtual Storage Integrator (VSI). I recommend reading the VSI USM Read Me First document to see what else needs to be installed to make the VSI plug-in work.

After the VSI USM plug-in and all the other needed packages have been installed the VNXe has to be connected to vCenter. This is done from vCenter Home – Solutions and Applications – EMC – Unified Storage Management – Add. A wizard will walk through the steps needed to connect vCenter and VNXe.

Now storage can be provisioned to a cluster or to an individual ESX host by right clicking cluster/host and selecting EMC – Unified Storage – Provision Storage.

The wizard follows the same steps as the VMware Storage Wizard when provisioning storage from VNXe.

Storage type:

Storage Array:

Storage Pool:

iSCSI Server:

Storage Details:

When using VSI to provision storage the iSCSI initiators and targets are configured automatically and VMFS datastore is also created in the process.

Conclusions

Again the suitable word to describe storage provisioning would be simple, if it would work every time. After provisioning several datastores I noticed that a VMFS datastore wasn’t always created when the iSCSI storage was provisioned from VNXe. Also there were issues if CHAP wasn’t used on VNXe but was used on ESX host for other datastores. This happens when using either VNXe or VSI storage provisioning.

Storage provisioning from VNXe is easy but it is even easier using VSI. When the initial setup is done, the iSCSI/NFS server configured and the storage pool(s) created there isn’t a need to login to VXNe anymore to provision storage if VSI is in use. This of course needs vCenter and all the necessary plug-ins to be installed.

Some users might never see these issues that I found out but for some these might be show stoppers. Not all businesses have vCenters in use so they have to use the Unisphere UI to provision storage and then the VMFS datastore might or might not be created. I can imagine how frustrated users can be when facing these kinds of issues.

Also, users shouldn’t  be responsible of the decision in which SP the new datastore is placed on. This should be something that VNXe decides.

Don’t take me wrong. The integration with VNXe and vCenter/ESX is smooth and it will be even better after these issues have been fixed.

In the next part of my hands-on series I will look into the performance of VNXe 3300 and I will also post some test statistics.


Hands-on with VNXe 3300 Part 4: Storage pools


When EMC announced that VNXe will also utilize storage pools my first thought was that it is similar to what CX/VNX has. Storage pool would consist of five disk RAID 5 groups and LUNs would be striped across all of these RAID groups to utilize all spindles. After some discussions with EMC experts I found out that this is not how the pool works in VNXe. In this part I will go a bit deeper into the pool structure and also explain how Storage Pool is created.

Hands-on with VNXe 3300 series [list edited 9/23/2011]:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

Pool Structure

VNXe 3300 can be furnished with SAS, NL-SAS or Flash drives. The one that I was configuring had 30 SAS disks so there were two options when creating Storage Pools: 6+1 drive RAID 5 groups or 3+3 RAID 1/0 groups. I chose to create one big pool with 28 disks (four 6+1 RAID 5 groups) and one hot spare disk (EMC recommends having one hot spare disk for every 30 SAS disks).  EMC also recommends not putting any I/O intensive load on the first four disks because PSL (Persistent Storage Layout) is located on those disks. I wanted to test the storage pool performance with all the disks that were available so I ignored this recommendation and also used the first four disks in the pool too.

When LUN is created it will be placed on the RAID group (RG) that is the least utilized from a capacity point of view. If the LUN created is larger than the free space on an individual RG the LUN will be then extended across multiple RGs but there is no striping involved. So depending of the LUN size and pool utilization a new LUN could reside either in one RG or several RGs. This means that only one RG is used for sequential workloads but random workload could be spread over several RGs. Now if disks are added to the storage pool those newly added RGs are the least utilized and will be used first when new LUNs are created. So a storage pool on VNXe can be considered more as a capacity pool than a performance pool.

Before I wrote this post I was in contact with EMC Technology Consultant (TC) and EMC vSpecialist to get my facts right. Both of them confirmed that the LUNs in VNXe pool are not striped across RGs. Pool structure was explained to me by the EMC TC. Looking at the test results that I posted on part 6 and also looking at the feedback that I got the description above is not accurate. Here is a quote from Brian Castelli’s (EMC employee) comment:

 “When provisioning storage elements, such as an iSCSI LUN, the VNXe will always stripe across as many RAID Groups as it can–up to a maximum of four.”

Based on Brian’s comment LUNs in VNXe pool are striped across multiple RGs. [Edited 9/15/2011]

Creating Storage Pools

Storage pools are configured and managed from System – Storage Pools. If no pools have been configured then Unconfigured Disk Pool is only shown.

Selecting Configure Disks will start disk configuration wizard and there are three options to select from: Automatically configure pools, Manually create a new pool, and Manually add disks to an existing pool. Quite easy to understand what each option stands for. I chose the Automatically configure pools option. When using the automatic configuration option 6+1 disk RAID 5 groups are used to create the pool.

Next step is to select how many disks are added to the new pool and you can see that the options are multiples of seven (6+1 RAID 5).

A hot spare pool will also be created when using the automatic pool configuration option.

When selecting Manually create a new pool there is a list of alternatives (see picture below) based on the desired purpose of the pool. This makes creating a storage pool easy because VNXe suggests the RAID level based on the selection that the user made. There is also an option further down on the wizard where the user can select the number of disks used and the RAID level (Balanced Perf/Cap R5 or High Performance R1/0).

Conclusions

It feels a little disappointing to find out that the pool structure wasn’t what I was expecting it to be. But maybe my expectations were also too high in the first place.

Creating a Storage Pool is in line with one of EMC’s definitions for VNXe: simple. When Automatic configuration option is selected Unisphere will take care of deciding what disks are used in the pool and what is the correct number of hot spares needed based on EMC’s best practices.

The next part will cover storage provisioning from VNXe and also using EMC’s VSI plug-in for vCenter.


Hands-on with VNXe 3300 Part 3: Software update


Software update, Operating Environment update or firmware update. Those are the most commonly used synonyms to describe the same thing: Software update. On supportbeta.emc.com they call it VNXe Operating Environment update. In Unisphere you can find “Update software”  and by clicking that you will see the current System software version. I’ll be talking about software update.

I was planning to post this as the last part of my hands-on experience series. But while I was writing the series I tested the latest software update (2.1.0) on other VNXe and found out there were fixes that would change some of the posts a lot. So I decided go through the software update at this point and write the remaining parts based on the 2.1.0 software version.

Hands-on with VNXe 3300 series [list edited 9/23/11]:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Upgrade to latest software version: new features and fixed “known issues”
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

Preparation

The first thing is to download the latest software update from support.emc.com. While downloading the file I opened a chat to EMC technical support just to make sure I was good to go with the update on production environment. The answer was no. Because our systems had been up and running for more than 42 days we would have to power down the whole VNXe before we could proceed with the update. Yes, power down dual SP storage system. Seriously? For a while I didn’t know what to think. I was wondering was it because we were running 2.0.3 or because of the 42+ day uptime and the technician confirmed that it was the 42+ days. This is the information that I got last week. It could be different this week, or with other sw version so check with support when planning to do an update.

So we had two VNXe 3300’s on production with over hundred VMs running on them and we had to take all of those VMs down. We also had one VNXe that was waiting to be configured. I decided to do the update on the new VNXe 3300 first and see if it was worth of taking the VMs down and updating the software. Because the new VNXe didn’t have any hosts connected to it yet it was pretty straight forward. In software update 2.1.0 there were big enough fixes and enhancements to Unisphere that I decided to proceed updating the software. I’ll later go through the reasons which lead to this decision.

Powering down

Before moving to the software update procedure itself the VNXe had to be powered off. The technician gave me 30 step instructions how to do this so here is a short version:

  • Stop all I/O to VNXe
  • Place both SPs in Service Mode
  • Disconnect power from DPE
  • Disconnect power from DAE
  • Reconnect power on DAE
  • Reconnect power on DPE
  • Reboot each SP to return to Normal Mode.
Placing SPs in Service More is done from Settings – Service System. Service account password is needed to access this area. Non-primary SP has to be set to Service Mode first. After the SP has rebooted itself and and the state of the SP changes from unknown to service mode then the the primary SP can be also set to Service Mode.
 
After the VNXe is powered back on then Unisphere only allows login with the service account. Now SP A has to be rebooted first which will return it to Normal Mode and after this SP B can be rebooted. This whole process takes about an hour and now VNXe is ready for the software update.
Updating software
Now that the update package is downloaded from supportbeta.emc.com it has to be uploaded to the VNXe. All the update steps can be done from Settings – More configurations – Update Software.
 

When the upload is ready Candidate Version changes from Not Available to 2.1.0. Before it can be installed a Health Check needs to be ran. If the Health Check doesn’t report any errors the software update can be installed by clicking Install Candidate Version.

Issues with the update

I did the update on four VNXe’s and two out of those had some issues with the update. The other VNXe that we had to power down before the update wasn’t that willing to boot the SPs on Service Mode. I followed the instructions that I got from technical support and after stopping the I/O I placed the non-primary SP (A) on Service Mode. SP A didn’t come back up after 20 minutes, not even after an hour. That time I started suspecting that there was something wrong. From Unisphere everything seemed to be normal: In Service System SP B seemed to be primary and in Normal mode and SP A showed an unknown state, which is normal during the reboot. I then found out that the management IP was not responding at all so Unisphere must have shown pages from the browser cache. There was a small indicator about Unisphere not working properly on the lower left corner of the window and also got error “Storage System Unavailable” when trying to execute any commands on VNXe:

The way that the lights were blinking on SP A indicated that it was on Service Mode. Unplugging the SP B management port the management IP started answering and I was able to login to Unisphere. Now Unisphere was showing that SP A was primary and that it was also in Service Mode. SP B was in Normal mode. I then placed SP B in Service Mode and proceeded powering off the VNXe. It seemed that SPs were in some kind of conflict mode. Software update itself went through without problems.

Another issue that I faced was during the software upgrade of VNXe loaded with NL-SAS disks. The update was halfway done when Unisphere gave an error that upgrade has failed:

After restarting the upgrade it went through without any problems.

However, I noticed that the PSL (VNXe OS runs from internal SSD but uses an area called PSL – Persistent Storage Layout  – located on the first four disks on DPE) was now located on the 1st, 2nd, 3rd and 5th disk on the DPE. My guess is that the failed software update caused this. I hope this is not going to cause problems later.

Before the update:

After the update:

Fixes and new features

The first noticable difference is that the Unisphere version was updated from 1.5.3 to 1.6.0.

One issue that I criticised in the first part was that none of the support functions were working on version 2.0.3. Those are now fixed in the version 2.1.0. For example clicking How To Videos under Support will open the browser at supportbeta.emc.com and will ask powerlink username and password. My impression was that all the support functions were supposed to be integrated to the Unisphere. Now it is just a link to the supportbeta site. But hey, at least it works now. If it only could use the stored EMC Support Cretentials.

Bigger fix was done to the block size that is used when creating VMware Datastore. In 2.0.3 when VMware Datastore was created and automatically provisoned to ESX server the vmfs partition was created with 1MB block size. This meant that the maximum file size on that vmfs was 256GB. This has been changed on the version 2.1.0 and 8MB block size is used when new VMware Datastore is created.

Nice new feature on System performance monitoring is that now also Network Activity and Volume Activity can be monitored. It used to only show the CPU Activity.  Timeframe can be viewed on a scale 48h/24/1h and in addition to those CPU Activity can also be viewed by activity over the last 5 minutes.

Conclusions

Overall the update procedure is easy and takes about an hour if it’s done on newly installed VNXe. If the system has been up and running for over 42 days then scheduled downtime is necessary to complete the update. I would schedule at least four hours of downtime. Powering down and updating can be done in two hours if everything goes without problems. It took me about 3.5 hours to complete the update with one of the VNXe’s. This is something that should be fixed soon. Isn’t the point of a dual SP storage system the ease of serviceability: the other SP can be updated/replaced while the other SP takes care of the I/O. Currently that is not the case.

VNXe is still a fairly new product and I would think that it’s still heavily developed and software updates will be released quite often. To fix issues and also to add some new features. So it’s good to keep an eye on the supportbeta.emc.com downloads section for the new releases. It is also important to contact EMC technical support before installing new updates on VNXe just to check that it’s ok to update or if there are some extra steps that need to be done. Of course I recommend also reading the release notes first.

In the fourth part I will look in to the storage pools and storage provisioning.


%d bloggers like this: