Hands-on with VNXe 3300 Part 6: Performance


Now that the VNXe is installed, configured and also some storage has been provisioned to ESXi hosts it is time to look at the performance. Like I mentioned in the first post I had already gathered some test results from CX4-240 using Iometer and I wanted to make similar tests with VNXe so that the results could be comparable.

Hands-on with VNXe 3300 series:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

Test environment CX4-240

  • EMC CX4-240
  • Dell PE M710HD Blade server
  • Two 10G iSCSI NICs with total of four paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O  paths
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Test environment VNXe 3300

  • EMC VNXe 3300
  • Dell PE M710HD Blade server
  • Two 10Gb iSCSI NICs with total of two paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O paths (see Trunk restrictions and Load balancing)
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Iometer Configuration

I used Iometer setup described in VMware’s Recommendations for Aligning VMFS Partitions (page 7) document.

Disk configuration

I had to shorten the explanations on the charts so here are the definitions:

  • CX4 FC 15D
    • 15 15k FC Disk RAID5 Pool on CX4-240 connected with iSCSI
  • CX4 SATA 25D
    • 25 7.2k SATA Disk RAID5 Pool on CX240 connected with iSCSI
  • VNXe 21D 2.0.3
    • 21 15k SAS Disk RAID 5 (3×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.0.3
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300. connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.1.0
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.1.0
  • VNXe RG 2.1.0
    • 7 15k SAS RAID5 (6+1) RG on VNXe connected with iSCSI. VNXe Software version 2.1.0
  • VNXe 28D NFS
    • 28 15k SAS RAID 5 (4×6+1) Pool on VNXe 3300 connected with NFS. VNXe Software version 2.1.0

100Gb thick LUN was created to each pool and RG where 20Gb virtual disk was stored. This 20Gb virtual disk was presented to the virtual Windows server that was used to conduct the test. Partition to this disk was created using diskpart command ’create partition primary align=1024′ and partition was formatted with a 32K allocation size.

Trunk restrictions

Before I go through the results I want to address the limitation with the trunk between VNXe and the 10Gb switch that it is connected to. Even though there are 4Gb (4x1Gb) trunk between the storage and the switch the maximum throughput is only the throughput of one physical port.

While I was running the tests I had SSH connection open to VNXe and I ran netstat -i -c command to see what was going on with the trunk and individual ports. The first screen capture is taken while 8k sequential read test was running. You can see that all the traffic is going through that one port:

The second screen capture is taken while the VNXe was in production and several virtual machines were accessing the disk. In this case the load is balanced randomly between the physical ports:

Load balancing

VNXe 3300 is active/active array but doesn’t support ALUA. This means that LUN can only be accessed through one SP. One iSCSI/NFS server can only have one IP and this IP can only be tied to one port or trunk. Also LUN can only be served by one iSCSI/NFS server. So there will be only one path from the switch to VNXe. Round robin path selection policy can be enabled on ESX side but this will only help to balance the load between the ESX NICs. Even without the trunk round robin can’t be used to balance the load between the four VNXe ports.

Test results

Each Iometer test was ran twice and the results are the average of those two test runs. If the results were not similar enough (i.e. several hundred difference in IOps) then a third test was ran and the results are the average of those three runs.

Same as previous but without NFS results:

Average wait time

Conclusions

The first thing that caught my eye was the difference between VNXe 28 disk pool and 7 disk RG on the random write test. Quote from my last post about the pool structure:

When LUN is created it will be placed on the RAID group (RG) that is the least utilized from a capacity point of view. If the LUN created is larger than the free space on an individual RG the LUN will be then extended across multiple RGs but there is no striping involved.

The tests were ran on 100Gb LUN so it should fit in one RG if the information that I got would be correct. So comparing the pool and the results from one RG random write test it seems that even smaller LUNs are divided across multiple RGs.

Another interesting detail is the difference of the software version 2.0.3 and 2.1.0. Looking at the difference of these results it is obvious that the software version has a big effect on the performance.

NFS storage performance with random write was really bad. But with the 1k sequential read it surprised giving out 30000 iops. Based on these test I would stick with the iSCSI and maybe look at the NFS again after the next software version.

Overall VNXe is performing very well compared to CX. With this configuration the VNXe it is hitting the limits of the one physical port. This could be fixed by adding 10Gb I/O modules. Would be nice run the same test with the 10Gb I/O modules.

We are coming to an end of my hands-on series and I’ll be wrapping up the series in the next post.

[Update 2/17/2012] Updated NFS performance results with VNXe 3100 and OE 2.1.3.16008: VNXe 3100 performance

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.

Advertisements

8 responses to “Hands-on with VNXe 3300 Part 6: Performance

  • Ken

    Hi Henri,

    A minor correction on load balancing.

    There are a few ways you can get two links worth of bandwidth from one iSCSI datastore.

    How you appear to have things configured now, ESX will create only one TCP connection from an ESX host to an iSCSI datastore. If you are using LACP then the load will all go onto one link as you observed.

    There are two parts to this. The connection from the VNXe to the switch and the connection from the ESX hosts to the switch.

    Since you are using LACP already, you don’t necessarily need to do anything on the VNXe side. As long as you make the required config changes on the ESX side required to create more connections to a datastore, LACP will distribute the connections over the available ports. You can add connections by either adding an additional IP interface on the VNXe or adding vmkernel interfaces on the ESX side.

    On the ESX side you say you have 10Gb NICs. You don’t say how many or how many vmkernel interfaces you have. If you only have one vmkernel interface you will need to add either 3 if you leave the ESX side alone or 1 if you add a second interface on the VNXe iSCSI server.

    ESX creates paths to a datastore from each vmkernel interface to each target IP. Remember that for ESX 4.x you need to create the vmkernel interfaces from the command line to configure jumbo frames.

    You will also need to do port bindings. Page 40 or so in the VMware 4.1 iSCSI SAN config.

    The last step to actually access the possible bandwidth to a single datastore is to configure a load balancing policy that make sense for the VNXe. By default, ESX sets the policy to fixed. Things work better if you set the policy to round robin and make a key change at the command line. The setting is described over in Chad Sakac’s blog.

    http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html

    Search the page for “iops” to get to the section you need.

    By default, NMP doesn’t do “true” round robin. It sends 1000 I/Os down one connection before moving on to the next link. You can change this setting to 1 and get true round robin behavior.

    If you do all this and you run a sequential workload that has more than one I/O outstanding at a time you should see more than one link active to a single datastore.

    • henriwithani

      Thank you for your comment. Good points there.

      On VNXe side I have one four port (1GB) trunk and on ESX side I have two 10Gb NICs, one vSwitch and two vmkernel interfaces for iSCSI. Port bindings are also done. RR is set but I didn’t change the default NMP setting. I’ve done some testing with it in the past and for some reason it always changed back to the default after ESX was rebooted. I might look in to that again.

      The tests were done with only one datastore and iSCSI server and the test server was the only one accessing the storage. After testing I ended up creating four 1.4TB datastores and also four iSCSI servers. Two datastores and iSCSI servers on SP A and two on SP B. I then dedicated one iSCSI server per datastore. Now I have seven hosts (each with two vmkernel interfaces) accessing the same datastores and I can see that the traffic on VNXe side is quite well distributed between the four ports in the trunk.

      • Ken

        ESX 4.0 update 1 had a bug with the iops setting changed to something other than the default. On a reboot it would change the value to some random large value. I think that’s fixed in 4.1 and later.

        How well traffic is distributed across the links in a trunk is also somewhat a matter of luck when the number of connections is small. Each end of an LACP trunk controls load balancing for the packets it transmits. On the VNXe end the driver we use put traffic on links based on a source MAC/destination MAC hash. The algorithm is:

        ( source MAC XOR destination MAC ) mod (number of links)

        For a 2 link group you have a 50% chance that a unique MAC pair will still hash to the same link. As the number of connected hosts goes up the better your odds are that things will distribute well.

        We don’t expose the ability to change the load balancing algorithm on the VNXe end but you can change it on some switches. Cisco gives you six choices.

  • Isaak

    Hi Henri,
    First thanks for the great series of blog. We are running on 2.0.3.13400 and NFS and have been experiencing performance problems with our box. Do you know if EMC has acknowledge anything or issued out any fixes? If not I may have to convert everything back to iSCSI and lose the replication feature to another VNXe box.

  • Jason

    My company recently purchased both a VNX 5300 and a VNXe 3100. So far I have only gotten the VNXe wired up and I have been doing tests on it. I am getting similar ratios on the performance difference between iSCSI and NFS. Of course, the EMC reps are saying that NFS provides equal performance on 10GB jumbo frame networks but I am not seeing it. We are using ESXi 5 which the rep said makes the world of difference. Staying with iSCSI on the VNXe isn’t a problem, but they VNX doesn’t have iSCSI without coughing up an additional 35% on the price.

    Would it be possible for you to try your test again but with ESXi 5?

  • EMC VNXe NFS Performance at a Brief Glance… « VeeBits – Virtualization and Infrastructure

    […] concern I had with choosing NFS was a bleak performance report for certain IOMeter tests on Henriwithani’s VNXe rundown. I had time to run a brief IOMeter […]

  • superkikim

    Hi Henri,

    Would you care to share your iometer.cfg file ?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: