Qualifying vSphere releases for use in production

If you have ever wondered when it time to upgrade VMware vSphere; you might want to consider the experience of the unfortunate and unwise early adopters of vSphere 5.1. The time to upgrade is clearly NOT the first public release of any new version of vSphere!

For example: vSphere 5.1 was plagued with numerous issues, including writing the main password for SSO in plaintext to a file on the vCenter server! vSphere 5.1 was pushed to release at VMworld 2012 for political reasons, plain and simple. Within the first three months, VMware released three major upgrades to vSphere 5.1, and even then certain critical features, like snapshot size alarms, never worked properly. My company never Qualified vSphere 5.1 for use in production and recommended that all of our customers hold at vSphere 5.0 until something better was available.

If you have the resources, the best-case scenario would be to install the latest version of vSphere and test all of the features you use in production. If any of the things you rely on do not work in the latest release, the conclusion is simple: It does not Qualify!

It is also important to pay attention to the blogosphere and the VMware communities , because issues like the plaintext passwords in vSphere 5.1 might not be immediately apparent in testing.

Many users do not have enough resources to test the latest vSphere release on the actual hardware used in production. In that case, building a virtual lab is a partial alternative to testing on production hardware. Unfortunately, virtual labs can not prove the functionality of vendor-customized ISOs of ESXi, nor network and storage issues that may impact your production vSphere environment.

My company utilizes less than 50% of our vSphere cluster at any given point in time. We have that defined in our SSAE 16 SCO1 Type 2 statement to auditors, and do so in order to fulfill the NIST requirement of rapid scalability in Cloud Computing. As we always have surplus physical hosts, it is easy to purpose several of them and install the latest vSphere release on the trial.

We test all of the features that we use against the HP VIB including:

• The vSphere Clients
• Networking
• Storage
• Veeam (and the vSphere API for Data Protection)

When VMware released vSphere 6, I began testing and noticed several small issues, but none that would prevent my company from Qualifying vSphere 6. It was only as a result of monitoring the Veeam forums that I realized CBT was broken and any given backup or replication that leveraged the vSphere API for Data Protection might be creating useless backup/replica images! This would have been true, not only for Veeam, but all backup applications which leveraged CBT. As a result, we did not Qualify vSphere 6 for use on our servers, and recommended all customers wait as well.

Another critical consideration is: how often is the version going to be updated? Early adopters of vSphere 5.1 should have upgraded vCenter 3 times before the end of the year (in three months!) in order to patch critical security and functionality issues. My company defines (as part of our SSAE 16 SOC 1 type 2) that vSphere releases must go without updates for 30 days before that version is suitable for use in production.

While vSphere 6.0.0 was unsuitable for use in production, we continued Qualifying that version and upgrading to each subsequent build. Eventually, vSphere 6.0.0b was released and went 30 days without updates, so we were able to Qualify that build and recommend it for use by our customers.

If your company does not have the resources to build the latest release of vSphere on production hardware, the best bet is to watch the blogasphere and VMware Communities for issues and make your decision based on the information you find.

I will continue to post the findings of my company (largely based on my own work, as the lead Solutions Architect) here for your reference. Although we use exclusively HP Blades, HP ProCurve Networking, and 10 Gb iSCSI SAN, many our our findings will apply equally to all environments. While HP Emulex issues are specific to HP, the CBT issue with vSphere 6.0.0 affected all users equally.