Category Archives: ESXi

VNXe 3300 performance follow up (EFDs and RR settings)


On my previous post about VNXe 3300 performance I introduced results from the performance tests I had done with VNXe 3300. I will use those results as a comparison for the new tests that I ran recently. In this post I will compare the performance difference with different Round Robin policy settings. I also had a chance to test the performance of EFD disks on VNXe.

Round Robin settings

On the previous post all the tests were ran on default RR settings which means that ESX would send 1000 commands through one path before changing the path. I observed that with the default RR settings I was only getting the bandwidth of one link on the four port LACP trunk. I got some feedback from Ken advising to change the default RR IO operation limit setting from 1000 to 1 to get two links worth of bandwidth from VNXe. So I wanted to test what kind of an effect would this change have on performance.

Arnim van Lieshout has a really good post about configuring RR using PowerCLI and I used his examples for configuring the IO operation limit from 1000 to 1. If you are not confident running the full PowerCLI scripts Arnim introduced in his post here is how RR settings for individual device could be changed using GUI and couple of simple PowerCLI commands:

1. Change datastore path selection policy to RR (from vSphere client – select host – configure – storage adapters – iSCSI sw adapter – right click the device and select manage paths – for path selection select Round Robin (VMware) and click change)

2. Open PowerCLI and connect to the server

Connect-VIServer -Server [servername]

3. Retrieve esxcli instance

$esxcli = Get-EsxCli

4. Change device IO Operation Limit to 1 and set Limit Type to Iops. [deviceidentifier] can be found from vSphere client’s iSCSI sw adapter view and is in format of naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

$esxcli.nmp.roundrobin.setconfig($null,”[deviceidentifier]”,1,”iops”,$null)

5. Check that the changes were completed.

$esxcli.nmp.roundrobin.getconfig(“[deviceidentifier]”)

Results 1

For these tests I used same environment and Iometer settings that I described on my Hands-on with VNXe 3300 Part 6: Performance post.

Results 2

For these tests I used the same environment except instead of virtual Win 2003 I used virtual Win 2008 (1vCPU and 4GB memory) and the following Iometer settings (I picked up these settings from VMware Community post Open unofficial storage performance thread):

Max Throughput-100%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 100% Read distribution
  • 5 minute run time

Max Throughput-50%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 50% read/write distribution
  • 5 minute run time

RealLife-60%Rand-65%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 40% sequential / 60% random distribution
  • 35 % read /65% write distribution
  • 5 minute run time

Random-8k-70%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 100% random distribution
  • 30 % read /70% write distribution
  • 5 minute run time
[Updated 11/29/11] After I had published this post Andy Banta gave me a hint on twitter:
You might squeeze more by using a number like 5 to 10, skipping
some of the path change cost.
So I ran couple of more tests changing the IO operation limit between 5-10. With the 28 disk pool there was no big difference when used the values 1 or 5-10. With the EFDs the magic number seemed to be 6 and with that I managed to get 16 MBps and 1100 IOps more out of the disks with specific work loads. I added the new EFD results to the graphs. 


Conclusions

Changing the default RR policy setting from the default 1000 io to 1 io really makes a difference on VNXe. On random workload there is not much difference between these two settings. But with sequential workload the difference is significant. Sequental write IOps and throughput is more than double with certain block sizes when using the 1 io setting. If you have ESXs connected to VNXe with LACP trunk I would recommend changing the RR policy to 1 5-10. Like I already mentioned Arnim has a really good post about configuring RR settings using PowerCLI. Another good post about multipathing is A “Multivendor Post” on using iSCSI with VMware vSphere by Chad Sakac.

Looking at the results it is obvious that EFD disks perform much better than SAS disks. On sequential workload 28 Disk SAS pool’s performance is about the same as 5 disk EFD RG’s. But on random workload EFD’s performance is about two times better than SAS pool’s. There was no other load on the disks while these tests were ran so under additional load I would expect EFD’s performing much better on sequential load as well. Better performance doesn’t come withouth a bigger price tag. EFD disks are still over 20 times more expensive per TB than SAS disks but then again SAS disks are about 3 times more expensive per IO than EFD disks.

Now if only EFDs could be used as cache on VNXe.

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.


VNXe Operating Environment 2.1.1


[Edited 10/12/2011] Apparently EMC has pulled back the VNXe OE MR1 SP1 (2.1.1.14913) and it is not available for download anymore. According to EMC new image and release notes will be available soon.      

EMC has released VNXe operating environment version 2.1.1.14913 (2.1 SP 1) for VNXe 3100 and 3300. And how did I find out about the new update? Well my VNXe told me about it:

Release notes and software are available on supportbeta.emc.com. The first thing that I noticed on the short description on the upgrade package was that VNXe OE has to be 2.1.0.14097 or higher before upgrading to 2.1.1. On the release notes I couldn’t find any mention about this. The only mention about mandatory upgrade is that VNXe should be upgraded to version 2.0.2 or later within 42 days of the initial VNXe installation or otherwise the system has to be powered off before upgrade (KB article emc265195). I also mentioned about this issue on my previous post about VNXe software update. So I contacted the support using chat and quickly got a confirmation that the VNXe has to be on 2.1.0.14097 before upgrading to 2.1.1.

Here is a quick pick of the new features, enhancements and fixes. Full descriptions can be found from the release notes.

New features and enhancements

6+1 RAID5 is now supported on VNXe 3100 with SAS drives and user-defined pools. Automatic configuration will still use 4+1 RAID5 for SAS drives.

EFD drives and 1TB NL SAS drives are now supported on VNXe 3300 DPE and DAE.

There have also been improvements to Unisphere performance.

Fixed

  • Looping problem that might cause SP to boot when network or power cable is disconnected
  • Issues with email alerts
  • Issues with password reset button causing SP to reboot
  • Error with hidden shared folders
  • VMFS datastore creation issues

There is also a long list of known problems and limitations. Couple of those concern VMware integration and are good to keep in mind:

  • VMFS datastore created from VNXe will be VMFS3 an use 8MB blocksize.
  • Manual rescan for storage is required after deleting datastore from standalone ESX server

Hands-on with VNXe 3300 Part 7: Wrap up


When I started writing this series my first plan was to just make four quick posts about my experience with EMC VNXe 3300. After I started going further into detail on the configuration I realized that I couldn’t get everything that I wanted to express to fit in four posts. So I ended up adding three more posts to the series. Still, with this series I’m just touching the surface of VNXe 3300. There are so many functionalities that I didn’t go through or didn’t even mention. That was also my intention with this series. I wanted to write about my experience and look in to those features that I would implement.

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

EMCWORLD_39

Simple, Efficient, Affordable

Those are the words that EMC uses for marketing VNXe and I can mostly agree that those are accurate adjectives for VNXe. In the first part I also wanted to add the adjective “easy” among those words. A user can do the initial setup and VNXe can be operational in less than an hour depending on the software version. Unisphere UI layout is also very user friendly and illustrative. Configuration and updating SW are easy and simple.

EMCWORLD_42

Customers buying VNXe just based on the marketing material might face a big surprise when looking in to the actual memory configuration. Yes, VNXe has 12GB memory per SP, but only 256MB is dedicated for read cache and 1GB dedicated for write cache.

Configuration considerations

Even though it is easy and simple to get VNXe up and running and to start provisioning storage this doesn’t mean that the planning phase can be ignored. User can easily be in a really bad situation and the only way out is to delete all datastores to do the proper reconfigurations. Creating only one iSCSI server and putting all datastores on that one server creates a situation where all the I/O goes through one SP and the other SP is idle. Depending on the ESX iSCSI settings only one network port on VNXe could be utilized even if a four port trunk was configured. Fixing this problem is not as easy as creating it. VNXe doesn’t allow changing the datastore iSCSI server after the datastore is created. To assign a different iSCSI server for a datastore it has to be deleted and recreated. This is, again, one issue that I’m hoping will be fixed.

When using four 1GB ports my suggestion would be to configure NIC aggregation on VNXe as I described in part 2. For the ESX configuration I would suggest reading the detailed comment Ken posted in part 6 about the ESX iSCSI configurations. What comes to the VNXe iSCSI and datastore configurations I ended up creating equal number of datastores for each SP and also dedicating one iSCSI server per datastore to get the most out of the four port trunk.

Issues

The issues that I faced during the configuration were mostly minor usability flaws and some of those were already fixed in the latest software version. The biggest issue that I found was that the VNXe had to be powered off before a software update if it had been running for more than 42 days. I’ve discussed these issues with EMC and hopefully they will be fixed in the future releases.

Conclusions

Despite all the criticism I think that VNXe 3300 is a great product and it will be even better when the few small flaws are fixed. I’m really looking forward to seeing what kind of new features will be introduced in the future software releases. Chad Sakac gave a hint on his blog post about FAST VP support coming in to VNXe at some point. He also mentioned that VAAI (file) and SRM support will be coming out still this year.

I can see some new VNXe 3300 related blog posts in my near future but I think it is time to close this series and keep the new posts separate. If you have any questions about my experience with VNXe or other general questions about it please leave a comment.


Hands-on with VNXe 3300 Part 6: Performance


Now that the VNXe is installed, configured and also some storage has been provisioned to ESXi hosts it is time to look at the performance. Like I mentioned in the first post I had already gathered some test results from CX4-240 using Iometer and I wanted to make similar tests with VNXe so that the results could be comparable.

Hands-on with VNXe 3300 series:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

Test environment CX4-240

  • EMC CX4-240
  • Dell PE M710HD Blade server
  • Two 10G iSCSI NICs with total of four paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O  paths
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Test environment VNXe 3300

  • EMC VNXe 3300
  • Dell PE M710HD Blade server
  • Two 10Gb iSCSI NICs with total of two paths between storage and ESXi. Round robin path selection policy enabled for each LUN with two active I/O paths (see Trunk restrictions and Load balancing)
  • Jumbo Frames enabled
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Iometer Configuration

I used Iometer setup described in VMware’s Recommendations for Aligning VMFS Partitions (page 7) document.

Disk configuration

I had to shorten the explanations on the charts so here are the definitions:

  • CX4 FC 15D
    • 15 15k FC Disk RAID5 Pool on CX4-240 connected with iSCSI
  • CX4 SATA 25D
    • 25 7.2k SATA Disk RAID5 Pool on CX240 connected with iSCSI
  • VNXe 21D 2.0.3
    • 21 15k SAS Disk RAID 5 (3×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.0.3
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300. connected with iSCSI. VNXe Software version 2.0.3
  • VNXe 28D 2.1.0
    • 28 15k SAS Disk RAID 5 (4×6+1) Pool on VNXe 3300 connected with iSCSI. VNXe Software version 2.1.0
  • VNXe RG 2.1.0
    • 7 15k SAS RAID5 (6+1) RG on VNXe connected with iSCSI. VNXe Software version 2.1.0
  • VNXe 28D NFS
    • 28 15k SAS RAID 5 (4×6+1) Pool on VNXe 3300 connected with NFS. VNXe Software version 2.1.0

100Gb thick LUN was created to each pool and RG where 20Gb virtual disk was stored. This 20Gb virtual disk was presented to the virtual Windows server that was used to conduct the test. Partition to this disk was created using diskpart command ’create partition primary align=1024′ and partition was formatted with a 32K allocation size.

Trunk restrictions

Before I go through the results I want to address the limitation with the trunk between VNXe and the 10Gb switch that it is connected to. Even though there are 4Gb (4x1Gb) trunk between the storage and the switch the maximum throughput is only the throughput of one physical port.

While I was running the tests I had SSH connection open to VNXe and I ran netstat -i -c command to see what was going on with the trunk and individual ports. The first screen capture is taken while 8k sequential read test was running. You can see that all the traffic is going through that one port:

The second screen capture is taken while the VNXe was in production and several virtual machines were accessing the disk. In this case the load is balanced randomly between the physical ports:

Load balancing

VNXe 3300 is active/active array but doesn’t support ALUA. This means that LUN can only be accessed through one SP. One iSCSI/NFS server can only have one IP and this IP can only be tied to one port or trunk. Also LUN can only be served by one iSCSI/NFS server. So there will be only one path from the switch to VNXe. Round robin path selection policy can be enabled on ESX side but this will only help to balance the load between the ESX NICs. Even without the trunk round robin can’t be used to balance the load between the four VNXe ports.

Test results

Each Iometer test was ran twice and the results are the average of those two test runs. If the results were not similar enough (i.e. several hundred difference in IOps) then a third test was ran and the results are the average of those three runs.

Same as previous but without NFS results:

Average wait time

Conclusions

The first thing that caught my eye was the difference between VNXe 28 disk pool and 7 disk RG on the random write test. Quote from my last post about the pool structure:

When LUN is created it will be placed on the RAID group (RG) that is the least utilized from a capacity point of view. If the LUN created is larger than the free space on an individual RG the LUN will be then extended across multiple RGs but there is no striping involved.

The tests were ran on 100Gb LUN so it should fit in one RG if the information that I got would be correct. So comparing the pool and the results from one RG random write test it seems that even smaller LUNs are divided across multiple RGs.

Another interesting detail is the difference of the software version 2.0.3 and 2.1.0. Looking at the difference of these results it is obvious that the software version has a big effect on the performance.

NFS storage performance with random write was really bad. But with the 1k sequential read it surprised giving out 30000 iops. Based on these test I would stick with the iSCSI and maybe look at the NFS again after the next software version.

Overall VNXe is performing very well compared to CX. With this configuration the VNXe it is hitting the limits of the one physical port. This could be fixed by adding 10Gb I/O modules. Would be nice run the same test with the 10Gb I/O modules.

We are coming to an end of my hands-on series and I’ll be wrapping up the series in the next post.

[Update 2/17/2012] Updated NFS performance results with VNXe 3100 and OE 2.1.3.16008: VNXe 3100 performance

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.


Hands-on with VNXe 3300 Part 5: Storage provisioning


When provisioning storage on VNXe there are many options depending on the host type: Microsoft Exchange, Shared folders, Generic iSCSI, VMware or Hyper-V . Because this VNXe is only going to serve VMware ESXi hosts I’m going to concentrate on that part. I will go through provisioning storage from VNXe and also using VSI Unified Storage Management Plug-in.

Hands-on with VNXe 3300 series [list updated 9/23/2011]:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Software update
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

iSCSI datastore

In the last part a storage pool was created. The next step is to create datastores on it and provision those to the hosts. There are two options when provisioning datastores for VMware from VNXe: VMware datastore or generic iSCSI. When using the VMware storage VNXe will automatically configure the iSCSI targets to selected ESX hosts and also create the VMFS datastore. If generic iSCSI is created then all those steps have to be done manually to each ESX. I really recommend using the VMware storage from these two options. VMware storage can be created from Storage – VMware storage.

At this point only iSCSI server was configured so the only option was to create a VMFS datastore.

The next step is to select the iSCSI server and the size for the datastore. When creating the first datastore it really doesn’t matter which iSCSI server is selected. If the iSCSI_A (on SP A) server is selected for the first datastore then iSCSI_B (on SP B) should be selected for the second datastore to balance the load on the SP’s. When selecting the iSCSI server on SP A this means that all the datastore I/O will go through the SP A. If all the datastores are placed on one SP there could be a situation where the whole VNXe performance is impacted because all I/O goes through that SP and the other SP is idle. So it is important to balance the datastores between SPs. VNXe is not doing this automatically so the user has to manually do this when creating datastores. If datastores are distributed between the SPs and the other SP fails all the datastores on the failed SP’s iSCSI server are moved to the other SP. When the failed SP comes back online all the datastores originally located on it are moved back.

There is an option to configure protection for the storage. Because this datastore is only for testing I chose not to configure the protection.

Step 5 is to select the hosts that the datastore will be attached to. Datastore can be connected to a specific iSCSI initiator on ESX server by expanding the host and selecting Datastore access for the specific IQN. If Datastore access is selected from the host level then the VNXe targets are added to all iSCSI initiators on ESX.

After these steps are completed VNXe starts creating the datastore, adding iSCSI targets to the selected ESX hosts and iSCSI initiators on those and finally mounts and creates the VMFS datastore.

 iSCSI datastore issues

After creating the first datastore I noticed from vCenter that there were “Add Internet SCSI send targets” and “Rescan all HBAs” tasks showing on the hosts that I selected to add the datastore to. After watching those tasks looping for 15 minutes and datastore not showing up on the ESXi servers I figured out that there was something wrong with the configuration.

I found out that the ESXi server had datastores connected also from other storage units that use CHAP authentication. On the ESXi iSCSI initiator the CHAP settings were set to “Inherit from parent” and that meant all the new targets would also inherit these CHAP settings. After disabling the inheritance the new datastore was connected and the VMFS datastore was created on it. I haven’t tried to use CHAP authentication with VNXe datastores so I don’t know if those settings are automatically configured to ESX. VNXe already has the ability to manipulate the ESX server configuration so I would imagine that it could also be possible to change the iSCSI target option to “Do not use CHAP” when VMware datastore is created on VNXe without CHAP authentication. Maybe in next the software version?

Another issue I had was that  a VMFS datastore was not always created during the process of creating a VMware datastore. VMware Storage Wizard on VNXe indicated that creating a VMware datastore is completed but actually the “create VMFS datastore” task was never initiated on the ESX. I’ve created over 20 datastores using this method and I would say that in about 50% of those cases the VMFS datastores were also created. Not a big thing, but still an annoying small glitch.

NFS datastore

Creating an NFS datastore is very similar to creating an iSCSI datastore. The only differences are steps 2 and 5 on the VMware Storage Wizard. On step 2 NFS is selected instead of VMFS (iSCSI). On this page there are two “hidden” advanced options: Deduplication and Caching. These options are hidden under the “Show advanced” link – similar to what criticized in iSCSI server configuration. In my opinion these should be shown by default.

On step 3 the same rule applies to selecting the NFS server as with the iSCSI server. The user has to manually balance the datastores between the SPs.

On step 5 datastore access (no access, read-only or read/write) will be chosen for the host or a specific ip address on host.

When all the steps on the wizard are done VNXe creates the datastore and mounts it to the selected host or hosts. I only gave one host access to this new NFS datastore but I could see that VNXe tried to do something NAS related to all the hosts in the Virtual Center connected to VNXe and gave some errors:

VSI

In a nutshell VSI Unified Storage Management (USM) is a plug-in that integrates with VMware vCenter and can be used to provision storage using vCenter UI. There is lots of good documentation on EMC Unified Storage Community and Powerlink so I’m not going to dig any deeper into it. VSI USM can be downloaded from Powerlink – Support – Software Downloads and Licensing – Downloads T-Z – Virtual Storage Integrator (VSI). I recommend reading the VSI USM Read Me First document to see what else needs to be installed to make the VSI plug-in work.

After the VSI USM plug-in and all the other needed packages have been installed the VNXe has to be connected to vCenter. This is done from vCenter Home – Solutions and Applications – EMC – Unified Storage Management – Add. A wizard will walk through the steps needed to connect vCenter and VNXe.

Now storage can be provisioned to a cluster or to an individual ESX host by right clicking cluster/host and selecting EMC – Unified Storage – Provision Storage.

The wizard follows the same steps as the VMware Storage Wizard when provisioning storage from VNXe.

Storage type:

Storage Array:

Storage Pool:

iSCSI Server:

Storage Details:

When using VSI to provision storage the iSCSI initiators and targets are configured automatically and VMFS datastore is also created in the process.

Conclusions

Again the suitable word to describe storage provisioning would be simple, if it would work every time. After provisioning several datastores I noticed that a VMFS datastore wasn’t always created when the iSCSI storage was provisioned from VNXe. Also there were issues if CHAP wasn’t used on VNXe but was used on ESX host for other datastores. This happens when using either VNXe or VSI storage provisioning.

Storage provisioning from VNXe is easy but it is even easier using VSI. When the initial setup is done, the iSCSI/NFS server configured and the storage pool(s) created there isn’t a need to login to VXNe anymore to provision storage if VSI is in use. This of course needs vCenter and all the necessary plug-ins to be installed.

Some users might never see these issues that I found out but for some these might be show stoppers. Not all businesses have vCenters in use so they have to use the Unisphere UI to provision storage and then the VMFS datastore might or might not be created. I can imagine how frustrated users can be when facing these kinds of issues.

Also, users shouldn’t  be responsible of the decision in which SP the new datastore is placed on. This should be something that VNXe decides.

Don’t take me wrong. The integration with VNXe and vCenter/ESX is smooth and it will be even better after these issues have been fixed.

In the next part of my hands-on series I will look into the performance of VNXe 3300 and I will also post some test statistics.


Running ESXi 5 on VMware Player


ESXi 5 was just released couple of hours ago and of course I had to try it. But I didn’t have any HW at home to run it on. So I saw @BlueShiftBlog’s tweet about running the ESXi 5 on VMware Player and I decided to give it a try. And it works.

Here is what you need to do:

When creating a new virtual machine on VMware Player “Other 64-bit” needs to be selected as the guest OS version. I used the default Hard Disk size but changed the memory to 2GB, added other network adapter and also added other CPU.

ESXi 5 installer starts automatically after the virtual machine is configured and powered on (assuming that you selected the VM to boot from the ESXi iso). You will get an error message “HW virtualization is not enabled” but that can be ignored.

After the installation is done and ESXi has rebooted you can connect to it using vSphere Client. If you have DHCP enabled on your network you will see the IP address of the ESXi on the console window. If not then you need to configure network settings from Customize System.

I’m now running two ESXi’s on my laptop and might be able to squeeze one more ESXi installation on it with 2GB memory. 2GB might be bit overkill for the purpose that I’m using those for at this point.

vSphere client looks very similar to the previous version. Next thing would be installing VM on one of the ESXi’s and then vCenter on it. Maybe I leave that for tomorrow.

Here is couple of other blogs for some more information about the new features:


Hands-on with VNXe 3300 Part 2: iSCSI and NFS servers


This is the second part in the series about my hands-on experience with EMC VNXe 3300. In the first part I described the initial setup of VNXe and also the challenges that I had during the setup. Before ESXi servers and virtual machines can access the storage there is still a couple of things that need to be done. In this post I will go through setting up network port aggregation, iSCSI server, NFS server and also how to connect ESXi hosts to VNXe.

Hands-on with VNXe 3300 series [list edited 9/23/11]:

  1. Initial setup and configuration
  2. iSCSI and NFS servers
  3. Upgrade to latest software version: new features and fixed “known issues”
  4. Storage pools
  5. Storage provisioning
  6. VNXe performance
  7. Wrap up

NIC Aggregation

This VNXe is furnished with eight (four/SP) 1GB NICs and is connected to ESXi hosts that are using 10GbE NICs for iSCSI. So all four NICs in each SP will be aggregated for maximum throughput. From each SP these four aggregated ports are then connected to separate switches where trunk is configured. These switches are connected to ESXi hosts with 10Gb uplinks.

NIC aggregation can be configured from Settings – More configurations – Advanced Configuration. Default MTU size is 1500 so if jumbo frames are enabled it needs to be changed to 9000. This needs to be done only for the first port, because the other ports are aggregated to this port.

Port aggregation is enabled by selecting “Aggregate with eth2” under the eth3 and hitting “Apply Changes”. After these settings are also done to eth4 and eth5 the aggregation is ready.

These settings need to be done only once and Unisphere will automatically configure both SPs using the same settings. This also means that SPs can’t have different aggregation settings.

iSCSI Server configuration

iSCSI server can be configured from Settings – iSCSI Server Settings. When creating the first iSCSI server the default storage processor will be SP A. When SP A already has an iSCSI server then SP B is automatically selected as the storage processor when creating the second server. Storage processor, Ethernet port and VLAN ID can be also changed under the Advanced settings.

NFS Server configuration

NFS server can be configured from Settings – Shared Folder Server Settings.
This is very similar to configuring an iSCSI server: there is only one additional step.

On “Shared Folder Types” page NFS and/or CIFS is selected. I was planning to do some testing only with NFS so I chose that.

Connecting ESXi hosts to VNXe

This VNXe will be connected to an existing vCenter/ESXi environment, so all the iSCSI settings are already in place on the hosts. VNXe will automatically discover all the ESX/ESXi hosts from vCenter. Only vCenter name or ip address and the appropriate credentials are needed. This makes things a lot easier. No need to manually register tens or hundreds of hosts. VMware hosts can be added to VNXe from Hosts – VMware.

When the discovery is done all ESX hosts will be shown under Virtualization Hosts. Also the total number of datastores are shown even if those datastores are not from the particular VNXe.

Hosts can be expanded in the view and all virtual machines on that ESX host will be shown. Also some interesting details are shown: OS type, state and associated datastore. Associated datastore is the name of the datastore as it is shown on ESX server. VNXe is pulling all this data from vCenter using the credentials provided earlier.

Even more details of an individual virtual machine can be viewed by selecting the VM and clicking Details.

Conclusions

Unisphere is really made to be easy and simple to use. Everything can be easily found from the menus and sub menus. Icons are big but are not just icons, there is also a subject line and an explanation. If this “Shared Folder Server Settings” would be the only info given then the real meaning of it might not be clear to everyone. But with the explanation it is very understandable:

I have small criticism about the first page of iSCSI and NFS server settings. I’m wondering why the advanced settings are hidden under the “Show advanced” link. The window is already big enough to have those settings shown in default. Ok, the page looks cleaner without the settings shown but it would really be more user friendly if they were not hidden. First time that I went through the settings I noticed the VLAN setting appeared on the summary page but I couldn’t remember seeing where to actually set it. So I went back to the first page and discovered the “Show advanced” link.

In the next part I will go through the software update procedure and look in to the issues that have been fixed.


MS iSCSI vs. RDM vs. VMFS


Have you ever wondered if there is a real performance difference when a LUN is connected using a Microsoft (MS) iSCSI initiator, using raw device mapping (RDM) or when the virtual disk is on VMFS? You might also have wondered if multipathing (MP) really makes difference. While investigating other iSCSI performance issues I ended up doing some Iometer tests with different disk configurations. In this post I will share some of the test results.

Test environment [list edited 9/7/2011]

  • EMC CX4-240
  • Dell PE M710HD Blade server
  • Two Dell PowerConnect switches
  • Two 10G iSCSI NICs per ESXi, total of four iSCSI paths.
  • Jumbo frames enabled
  • ESXi 4.1U1
  • Virtual Win 2008 (1vCPU and 4GB memory)

Disk Configurations

  • 4x300GB 15k FC Disk RAID10
    • 100GB LUN for VMFS partition (8MB block size)
      • 20GB virtual disk (vmdk) for VMFS and VMFS MP tests
    • 20GB LUN for MS iSCSI, MS iSCSI MP, RDM physical and RDM virtual tests

MS iSCSI initiator used virtual machine’s VNXNET 3 adapter (one or two depending on the configuration) that was connected to the dedicated iSCSI network through ESXi’s 10GB nic. MS iSCSI initiator multipathing was configured using Microsoft TechNet Installing and Configuring MPIO guide. Multipathing for RDM and VMFS disks was configured by enabling round robin path selection policy. When multipathing was enabled there were two active paths to storage.

Iometer configuration

When I was trying to figure out what would be the best way to test the different disk configurations I found a post “Open unofficial storage performance thread”  from VMware Communities. In the thread there is this Iometer configuration that would test maximum throughput and also simulate real life scenario. Other Community users have also posted their results there. I decided to use the Iometer configuration posted on the thread so that I could also compare my results with the others.

Max Throughput-100%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 100% Read distribution
  • 5 minute run time

Max Throughput-50%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 32KB transfer request size
  • 100% sequential distribution
  • 50% read/write distribution
  • 5 minute run time

RealLife-60%Rand-65%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 40% sequential / 60% random distribution
  • 35 % read /65% write distribution
  • 5 minute run time

Random-8k-70%Read

  • 1 Worker
  • 8000000 sectors max disk size
  • 64 outstanding I/Os per target
  • 500 transactions per connection
  • 8KB transfer request size
  • 100% random distribution
  • 30 % read /70% write distribution
  • 5 minute run time

Test results

Each Iometer test was ran twice and the results are the average of those two test runs. If the results were not similar enough (i.e. several hundreds difference in IOps) then a third test was ran and the results are the average of those three runs.

Conclusions

Looking at these results VMFS was performing very well with both single and multipath. Both RDM disks with multipathing are really close to the performance of VMFS. And then there is MS iSCSI initiator that gave kind of conflicting results. You would think that multipathing would give better results than single path, but actually that was the case only on the max throughput test. Keep in mind that these tests were ran on a virtual machine that was running on ESXi and that the MS iSCSI initiator was configured to use virtual nics. I would guess that Windows Server 2008 running on a physical server with MS iSCSI initiator would give much better results.

Overall VMFS would be the best choice to put the virtual disk on but it’s not always that simple. Some clustering softwares don’t support virtual disks on VMFS and then the options are RDM or MS iSCSI. There could also be limitations for physical or virtual RDM disk usage.

Disclaimer

These results reflect the performance of the environment that the tests were ran in. Results may vary depending on the hardware and how the environment is configured.


EFD vs. FC Pools


Our CX4 with Flare30 has been in production for about six months now and we decided to add some more FAST Cache on it. It currently has two mirrored 100GB EFDs configured as FAST cache and we just got two new 100GB disks to be added to the cache. We’ve also been pondering if we should add EFDs to the current pools for databases. Before adding the two new disks to the cache I wanted to make some performance tests on the EFDs. I also wanted to compare the EFD performance with the performance of the current pools that we have in production.

The focus of these tests was to see if the EFDs would have the desired performance advantage against the current pools that we already have in use. Like I mentioned we already have 100GB FAST cache in use and it is also enabled on the pools that I used to run these tests.

I used Iometer to generate the load and to gather the results. In the past I’ve done Iometer tests with storage arrays that are not in any other use. In those cases I’ve used iometer setup described in VMware’s Recommendations for Aligning VMFS Partitions document. Using those settings to run Iometer tests would have been time consuming and would have also generated huge load on the production CX. Now that I was only focusing to compare the simulated database load on different disk configurations I decided to run the test with only one transfer request size.

While I was creating the disks for the test I decided to add a couple of more disks and run some additional tests. I was curious to see how a properly alligned disk would really perform compared to an unaligned one and also what kind of performance difference there was between VMFS and RAW-disks. Yes I know that the VMware’s document I mentioned above already proves that an aligned disk performs better than an unaligned. I just wanted to know what was the case in our environment.

Test environment [list edited 9/7/2011]

  • CX4-240 with 91GB FAST Cache
  • Dell PE M710HD Blade server
  • 2x Dell PowerConnect switches
  • Two 10G iSCSI NICs with total of four paths between storage and ESXi. Round robin path selection policy enabled for each lun with two I/O active paths.
  • ESXi 4.1U1
  • Virtual Win 2003 SE SP2 (1vCPU and 2GB memory)

Disk Configurations

  • 15 FC Disk RAID5 Pool with FAST Cache enabled
    • 50GB LUN for VMFS partition
      • 20GB unaligned virtual disk (POOL_1_vmfs_u)
      • 20GB aligned virtual disk (POOL_1_vmfs_a)
    • 20GB LUN for unaligned RAW disk (POOL_1_raw_u)
    • 20GB LLUN for aligned RAW disk (POOL_1_raw_a)
  • 25 FC Disk RAID5 Pool with FAST Cache enabled
    • 50GB LUN for VMFS partition
      • 20GB unaligned virtual disk (POOL_2_vmfs_u)
      • 20GB aligned virtual disk (POOL_2_vmfs_a)
    • 20GB LUN for unaligned RAW disk  (POOL_2_raw_u)
    • 20GB LLUN for aligned RAW disk  (POOL_1_raw_a)
  • 2 EFD Disk RAID1 RAID Group
    • 50GB LUN for VMFS partition
      • 20GB unaligned virtual disk (EFD_vmfs_u)
      • 20GB aligned virtual disk (EFD_vmfs_a)
    • 20GB LUN for unaligned RAW disk (EFD_raw_u)
    • 20GB LLUN for aligned RAW disk (EFD_raw_a)

Raw disks were configured to use physical compatibility mode on ESXi.

Unaligned disks were configured using Windows Disk Management and formatted using default values.

Partitions to aligned disks were created using diskpart command ‘create partition primary align=1024′ and partition were formatted with a 32K allocation size.

Iometer configuration

  • 1 Worker
  • 8KB transfer request size
  • Read/write ratio of 66/34 and 100% random distribution
  • 8 outstanding I/Os per target
  • 4 minute run time
  • 60 sec ramp-up time

Results

Each Iometer test to specific disk was repeated three times. Results are the average of these three runs. Keep in mind that the array was running over 100 production VMs during the tests, so these results are not absolute.

Conclusions

When comparing the results on unaligned disks and aligned disks there are no huge differences. Although POOL_1_raw_u and POOL_2_vmfs_u results kind of jump out from those charts. I did three more test runs for those disks and still got the same results. This might have something to do with the production load that we are having on the CX.

Also the performance differences between raw disks and disks on vmfs were not major, but still noticeable, i.e. the difference on IOps between POOL_2_vmfs_a and POOL_2_raw_a is over 200. EFD raw is also giving about 200 more IOps than EFD vmfs.

Let’s get to the point. The whole purpose of these tests was to compare FC pool and EFD performance. If you haven’t noticed from the graphs the difference is HUGE! Do I even have to say more? I think the graphs have spoken. Those 5000+ IOps was achieved only with two EFDs. Think about having a whole array full of those.

After these tests my suggestion is to use VMFS datastores instead of raw disks. But there are still some cases that you might need to use raw disks with virtual machines, i.e. when having a physical/virtual cluster. Aligning Windows Server disks is not a big thing anymore because Windows Server 2008 does that automatically. If you have some old Windows Server 2003 installations I would suggest you to check if the disks are aligned or not. There is a Microsoft KB that describes how to check disk alignment. If the server disks are not aligned you might want to start planning to move your data to aligned disks. What comes to the EFDs the performance gained using those is self-evident. EFDs are still a bit expensive. But think about the price of the arrays and the disks needed for the same IOps than what the EFDs can provide. In some cases you need to think more about the price per IO than price per GB.

Disclaimer

These results reflect the performance of the environment that the tests were ran. Results may vary depending on the hardware and how the environment is configured.


%d bloggers like this: