Upgrading vRealize Operations

vRealize Operations (formerly known as vCenter Operations Manager) is an indispensable tool for monitoring a VMware environment. The upgrade process between recent versions is riddled with pitfalls, however. Here, I will cover the process and make notes of all the gotchas so that you can have as smooth of an experience as possible.

The upgrade process goes from 5.x -> 5.8.1+ -> 6.0.x -> 6.2.

Upgrading from 5.x to 5.8.5

Prerequisites and Common Pitfalls:

• You should change your admin password before attempting the upgrade. New security policies that come with this upgrade will expire older passwords for admin. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358


• You should ensure that you have a good amount of free disk space on the UI and Analytics VMs. You can check this with “df -h” on the command line.
  
  
  
  • Appliances that were originally deployed as 5.6 and earlier had a smaller root partition that may get filled by the update. If you run into any issues caused by this, you can follow http://kb.vmware.com/kb/2074688
  
  
  • If the /data partition is >85% full on either VM, you should add a disk and reboot. Boot-time scripts will handle data volume expansion onto new disks for you.
  
  
  
  

Steps to Upgrade:

1. Take a snapshot of both VMs.


2. Follow the instructions in the release notes for the upgrade at https://www.vmware.com/support/vcops/doc/vcops-585-vapp-release-notes.html#upgrade


3. After ensuring that you can get in to the UI, delete the snapshots.

Migrating data from 5.8.1+ to 6.0.x

Prerequisites and Common Pitfalls:

• You must be migrating from version 5.8.1 or higher. The data migration will be more reliable with version 5.8.5, but is known to work with all versions from 5.8.1+.


• You must have forward and reverse DNS entries in place for your source 5.8.1+ VMs, as well as for every node in your new 6.0.x cluster.


• Please size your nodes appropriately per the handy Excel spreadsheet attached to the bottom of http://kb.vmware.com/kb/2130551


• When naming and addressing your version 6 nodes, please note:
  
  
  
  • Node and host names can not have underscores
  
  
  • The roles of a node are subject to change, so naming a node according to a role may get confusing in the future.
  
  
  • Node names are extremely difficult to change, and attempting to do so is quite likely to break things.
  
  
  
  


• All nodes of a vROps 6 cluster (with the exception of remote collectors) must be on the same physical LAN (>1MS latency will cause problems) and must not be separated by a firewall.


• You should change your admin password on 5.8.1+ before attempting the migration. An expired password is common, sometimes difficult to identify, and will cause vague errors. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358


• If you suspect that performance may be an issue, stopping DT calculation and new data collection on the 5.8.1+ appliance will improve the speed of the data migration greatly. http://kb.vmware.com/kb/2040008


• Although it is not supposed to be an issue anymore, there are some cases where DNS resolution does not work properly and this KB may still be necessary: http://kb.vmware.com/kb/2100944

Steps to upgrade:

1. Install the latest version of 6.0.x per the documentation at http://pubs.vmware.com/vrealizeoperationsmanager-6/index.jsp


2. After you bring your cluster online, you will be presented with a wizard. Don’t select the option for importing data or enter a license key when it asks you. Even though you are going to import data and probably have a license key, it is more reliable to do these things after the setup wizard completes in my experience.


3. Go to Administration>Solutions and select the tab called “Import Data”


4. Follow the prompts for importing data.


5. Once the import is complete, you can run the old 5.8 instance in parallel with 6.0 until you are comfortable with the results, then delete the 5.8 instance when you are ready.

Upgrading from 6.x to 6.2

Prerequisites and Common Pitfalls:

• Because the upgrade process will convert two of the databases to another type and not delete the source data sets (in case of issues), you must plan for this extra disk space to be consumed.
  
  
  
  • To calculate how much extra space will be consumed by the database conversion, you can log in to the shell of each data/master/replica node and run:
    $  du -sch $STORAGE/db/vcops/*xdb* | tail -n1
    You should add around 10% to this value to be safe, which is in addition to the 15% total free space you ought to maintain on /storage/db for general stability.
  
  
  • To check how much total free space is available on the system, run:
    $ df -h $STORAGE/db
  
  
  
  


• If you have nodes that are separated by slow, latent, or unreliable links, the update may time out when the master node pushes the update out to them. This will be apparent if the upgrade fails without presenting you with an EULA. You can pre-stage the update pak files to work around this: http://kb.vmware.com/kb/2127895


• There are a few instances where upgrading to 6.2 or 6.2a will take an indefinite amount of time to complete. If your upgrade takes more than 24 hours to complete, please contact VMware support. They will be able to help finish the upgrade.
  This is fixed in 6.2.1.


• These should no longer be necessary, but I'll leaving them here for reference:
  
  
  
  • Previous updates, patches and add-on Management Packs may cause issues if their files still linger in your update queue. To avoid issues with this, you should log in to each node via console or SSH and move the pak files in “$STORAGE/db/casa/pak/dist_pak_files/” somewhere else.
  
  
  • Large database blobs from a previous data migration may choke the update process. http://kb.vmware.com/kb/2132479
  
  
  • The cluster state files may have some malformed data in them. http://kb.vmware.com/kb/2106950
  
  
  • Expired projects may cause an upgrade failure when going from 6.1 to 6.2 http://kb.vmware.com/kb/2136536
  
  
  • There may be a malformed upgrade state file from previous versions. You should remove this before upgrading. http://kb.vmware.com/kb/2120616
  
  
  • I have a script that can automate many of the above tasks and more at
    https://github.com/nakedhitman/vrops-cluster-repair
  
  
  
  

Steps to upgrade:

1. Take a snapshot of all your nodes.


2. Install the OS Update pak file. (This step is very important and must be done before the application update.)


3. Install the Application Update pak file.
   
   
   
   1. Ensure that you check both the boxes so that the “Reset out of the box content” option is selected. If you do not do this, some parts of the system may not be upgraded. This will not affect custom dashboards or other user-created content. Only the content that ships with the system that you ought not to have changed will be affected. If you do have customizations to the built-in content, you can clone them to preserve your changes.
   
   
   
   


4. Log in to the product UI as admin and ensure that your dashboards, adapter instances, and other data are present and working as expected.
   
   
   
   1. If it makes you go through the first run setup wizard again: don’t panic, its probably fine. Just choose evaluation mode and complete the wizard. Everything should be there when it’s complete. http://kb.vmware.com/kb/2132452
   
   
   2. If you end up with vCenter adapter instances that cannot be configured, missing dashboards or licenses, or end up loosing historical data, you should revert your snapshot and tell VMware about it immediately. If you don’t have a support contract, then make some noise on the communities site. VMware pays attention to this stuff and wants to prevent it from happening.
   
   
   
   


5. Once you’re sure that everything is running OK, delete the snapshots.

Conclusion

Although the upgrade process is a bit of an ordeal that can consume a good chunk of time, the features, stability, and performance of a successful upgrade really are worth it. VMware is working hard to make it better, and does listen to the things that are said on the communities site should you have any issues.