My Experiences Updating VCSA 6.5a to 6.5 U2

Today we'll be updating my lab's VCSA 6.5a (build 4944578) to version 6.5 Update 2 (build 8307201) which was released  May 3 2018. vCenter Server 6.5 U2 includes some backported features of vSphere 6.7 making it a slick update for users not quite ready to move to version 6.7. Some of the new features include

• Enhanced Linked Mode (ELM) for Embedded Platform Service Controller Architecture
• Backup & Restore to ELM with replication deployment topology API
• vMotion and cold migration of VMs across vCenters 6.0 U3+
• And more: https://blogs.vmware.com/vsphere/2018/05/vsphere-6-5-update-2-now-available.html

NOTE: Currently, upgrading from vSphere 6.5 U2 to vSphere 6.7 GA is not supported

You can verify your VCSA version by clicking on the top most level of the hierarchy tree in the vSphere Web Client and viewing the Summary tab

Before I perform this update, I'm going to verify compatibility with my registered and connected solutions with the VMware Interoperability Matrix. 

Then, I'm going to login to the vCenter Server Appliance Management Interface (VAMI) at <https://fqdn-of-vc:5480and ensure overall Health Status is green, take a VM-level snapshot (not pictured) and create a native VCSA Backup.

Upon entering my Backup details and clicking Next, I got a message stating that the service 'statsmonitor' wasn't running

A quick SSH session to the VCSA and running service-control --start vmware-statsmonitor resulted in starting the service

This allowed the Backup to continue

After it validated the input, we're ready to click Finish, which initiated the backup and showed a Backup Progress Bar

With our VM snapshot and VCSA Backup in place, we're ready to go to the Update section of the VAMI and check for Updates

Click Install Updates and accept the EULA to begin the update. My update jumped straight to 40% with status "Start staging packages" then had a "Download Failed"

At this point, I took a break for the evening (hey can't a man eat and unwind with some SNES Classic?!) and revisited it in the morning. This time, I did a tail -f on /var/log/vmware/applmgmt/software-packages.log to watch the progress of the downloads and to my surprise, the downloads worked this time without any changes made. If you run into a similar error, I would review this log to see what could be causing the downloads to fail (proxy config, incorrect URL, etc).

The Update window status changed to "Start running pre-install scripts"

Then made its way to 100% with the message "Packages upgraded successfully, reboot required to complete the installation"

And checking the software-packages.log we can confirm the packages were properly staged and ready for reboot

Clicking OK, I proceeded to the Summary Tab where I could click Reboot

Upon logging back in to the VAMI, we can verify the new build version!

Confirming in vSphere Web Client

Remember, we haven't fully upgraded to vSphere 6.5 U2 until the ESXi hosts are also updated. The embedded VUM instance in VCSA 6.5 makes that an almost effortless task.

Thanks for reading and good luck with your updates!

Read More

Horizon 7 version 7.0.2 Now Available

VMware Horizon 7.0.2 is here with a number of fixes and new features. I will document the upgrade process of my 7.0.0 lab in a separate post, including the Linux Desktops (which haven't been updated since the unmanaged process from Horizon 6) from unmanaged to managed desktops.

>>> Release Notes <<<
>>> Downloads <<<
>>> Official Announcement <<<

Horizon 7.0.2 Fixes

• View Administrator displays the error "Unable to query the replication state of the server" regarding the replication status of a Connection Server machine. This problem is a display issue that occurs with non-EN language Connection Server machines.


• The error message "No valid certificates were found on this smart card" appears on the Horizon Client for Windows, when you use a PIV smart card to authenticate to Windows 8 or Windows Server 2012 R2 and above desktops that do not have middleware installed. This problem is resolved when you install Horizon Agent 7.0.2 in the remote desktops and enable Windows auto update to let Windows install the PIV smart card minidriver on both the Windows client and Horizon agent.


• An error message appears when you run an executable (.exe) file that requires the administrator privilege from a network share folder on a physical workstation or virtual machine that has VMware View Persona Management version 6.2 installed.


• The Horizon Administrator Web page shows a page cannot be displayed error if you install View Connection Server 7.0.1 with the FIPS mode enabled on any drive other than the C drive.


• Windows 10 desktops that have Horizon 7.0 installed with View Persona Manager and some applications crash with a Blue Screen of Death.


• Windows Media MMR does not work on Windows 10 desktops.


• View Composer performs unsealed LDAP searches.


• In Horizon Administrator, when you add an instant-clone domain administrator and then create an instant-clone desktop pool, some virtual machines in the instant-clone desktop pool might fail to clone.


• In a Microsoft Network Load Balancer environment, an authentication error occurs when you use VMware Identity Manager to launch a remote desktop.

Horizon 7.0.2 New Features

• VMware Blast policy
  View administrators can configure the H.264 Quality Blast policy settings to specify the image quality for the remote display configured to use H.264 encoding.


• Client Drive Redirection
  For Horizon Client 4.2 or Horizon 7 version 7.0.2 or later, if VMware Blast Extreme is enabled, files and folders are transferred across a virtual channel with encryption.


• Global data collected by VMware
  If you join the customer experience improvement program, VMware collects data aboutTrue SSO for the View environment.


• Restrict remote desktop access outside the network
  View administrators can allow access to specific entitled users and groups from an external network while restricting access to other entitled users and groups.


• Smart Cards for multiple user accounts
  In some environments, a user's smart card certificate can map to multiple Active Directory domain user accounts. A user might have multiple accounts with administrator privileges and needs to specify which account to use in the Username hint field during smart card login. To make the Username hint field appear on the Horizon Client login dialog box, the administrator must enable the smart card user name hints feature for the Connection Server instance in View Administrator. The smart card user can then enter a user name or UPN in the Username hint field during smart card login.


• Remote desktop operating systems
  The following remote desktop operating systems are supported:
  
  
  
  • Windows 10 64-bit and 32-bit Enterprise and Professional; Build 1511
  
  
  • Windows 10 64-bit and 32-bit Enterprise Long Term Service Branch (LTSB); Fresh installation only; Build 1507
  
  
  • Windows 10 64-bit and 32-bit Anniversary Update; Fresh installation only; Tech preview only; Build 1607. Upgrade is not supported.
  
  
  
  


• Windows Media Multimedia Redirection (MMR) operating systems
  The following Windows MMR operating systems are supported:
  
  
  
  • Windows 10
  
  
  • Windows Server 2016 is a tech preview feature
  
  
  
  


• Windows registry keys for Flash Redirection
  View administrators can set requireIECompatibility=true to enable Flash Redirection for Web sites that support HTML5 by default. This parameter is not required for the YouTube Web site. In certain situations, setting appMode=0 can improve performance, and setting appMode=1 can result in a better user experience.


• Horizon 7 for Linux desktops enhancements
  
  
  
  • Automated full-clone desktop pool
  
  
  • SSO for SLED 11 SP3 and SP4
  
  
  • Horizon Client 4.2.0 for Android
  
  
  
  

 

 

Read More

Upgrading vRealize Operations

vRealize Operations (formerly known as vCenter Operations Manager) is an indispensable tool for monitoring a VMware environment. The upgrade process between recent versions is riddled with pitfalls, however. Here, I will cover the process and make notes of all the gotchas so that you can have as smooth of an experience as possible.

The upgrade process goes from 5.x -> 5.8.1+ -> 6.0.x -> 6.2.

Upgrading from 5.x to 5.8.5

Prerequisites and Common Pitfalls:

• You should change your admin password before attempting the upgrade. New security policies that come with this upgrade will expire older passwords for admin. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358


• You should ensure that you have a good amount of free disk space on the UI and Analytics VMs. You can check this with “df -h” on the command line.
  
  
  
  • Appliances that were originally deployed as 5.6 and earlier had a smaller root partition that may get filled by the update. If you run into any issues caused by this, you can follow http://kb.vmware.com/kb/2074688
  
  
  • If the /data partition is >85% full on either VM, you should add a disk and reboot. Boot-time scripts will handle data volume expansion onto new disks for you.
  
  
  
  

Steps to Upgrade:

1. Take a snapshot of both VMs.


2. Follow the instructions in the release notes for the upgrade at https://www.vmware.com/support/vcops/doc/vcops-585-vapp-release-notes.html#upgrade


3. After ensuring that you can get in to the UI, delete the snapshots.

Migrating data from 5.8.1+ to 6.0.x

Prerequisites and Common Pitfalls:

• You must be migrating from version 5.8.1 or higher. The data migration will be more reliable with version 5.8.5, but is known to work with all versions from 5.8.1+.


• You must have forward and reverse DNS entries in place for your source 5.8.1+ VMs, as well as for every node in your new 6.0.x cluster.


• Please size your nodes appropriately per the handy Excel spreadsheet attached to the bottom of http://kb.vmware.com/kb/2130551


• When naming and addressing your version 6 nodes, please note:
  
  
  
  • Node and host names can not have underscores
  
  
  • The roles of a node are subject to change, so naming a node according to a role may get confusing in the future.
  
  
  • Node names are extremely difficult to change, and attempting to do so is quite likely to break things.
  
  
  
  


• All nodes of a vROps 6 cluster (with the exception of remote collectors) must be on the same physical LAN (>1MS latency will cause problems) and must not be separated by a firewall.


• You should change your admin password on 5.8.1+ before attempting the migration. An expired password is common, sometimes difficult to identify, and will cause vague errors. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358


• If you suspect that performance may be an issue, stopping DT calculation and new data collection on the 5.8.1+ appliance will improve the speed of the data migration greatly. http://kb.vmware.com/kb/2040008


• Although it is not supposed to be an issue anymore, there are some cases where DNS resolution does not work properly and this KB may still be necessary: http://kb.vmware.com/kb/2100944

Steps to upgrade:

1. Install the latest version of 6.0.x per the documentation at http://pubs.vmware.com/vrealizeoperationsmanager-6/index.jsp


2. After you bring your cluster online, you will be presented with a wizard. Don’t select the option for importing data or enter a license key when it asks you. Even though you are going to import data and probably have a license key, it is more reliable to do these things after the setup wizard completes in my experience.


3. Go to Administration>Solutions and select the tab called “Import Data”


4. Follow the prompts for importing data.


5. Once the import is complete, you can run the old 5.8 instance in parallel with 6.0 until you are comfortable with the results, then delete the 5.8 instance when you are ready.

Upgrading from 6.x to 6.2

Prerequisites and Common Pitfalls:

• Because the upgrade process will convert two of the databases to another type and not delete the source data sets (in case of issues), you must plan for this extra disk space to be consumed.
  
  
  
  • To calculate how much extra space will be consumed by the database conversion, you can log in to the shell of each data/master/replica node and run:
    $  du -sch $STORAGE/db/vcops/*xdb* | tail -n1
    You should add around 10% to this value to be safe, which is in addition to the 15% total free space you ought to maintain on /storage/db for general stability.
  
  
  • To check how much total free space is available on the system, run:
    $ df -h $STORAGE/db
  
  
  
  


• If you have nodes that are separated by slow, latent, or unreliable links, the update may time out when the master node pushes the update out to them. This will be apparent if the upgrade fails without presenting you with an EULA. You can pre-stage the update pak files to work around this: http://kb.vmware.com/kb/2127895


• There are a few instances where upgrading to 6.2 or 6.2a will take an indefinite amount of time to complete. If your upgrade takes more than 24 hours to complete, please contact VMware support. They will be able to help finish the upgrade.
  This is fixed in 6.2.1.


• These should no longer be necessary, but I'll leaving them here for reference:
  
  
  
  • Previous updates, patches and add-on Management Packs may cause issues if their files still linger in your update queue. To avoid issues with this, you should log in to each node via console or SSH and move the pak files in “$STORAGE/db/casa/pak/dist_pak_files/” somewhere else.
  
  
  • Large database blobs from a previous data migration may choke the update process. http://kb.vmware.com/kb/2132479
  
  
  • The cluster state files may have some malformed data in them. http://kb.vmware.com/kb/2106950
  
  
  • Expired projects may cause an upgrade failure when going from 6.1 to 6.2 http://kb.vmware.com/kb/2136536
  
  
  • There may be a malformed upgrade state file from previous versions. You should remove this before upgrading. http://kb.vmware.com/kb/2120616
  
  
  • I have a script that can automate many of the above tasks and more at
    https://github.com/nakedhitman/vrops-cluster-repair
  
  
  
  

Steps to upgrade:

1. Take a snapshot of all your nodes.


2. Install the OS Update pak file. (This step is very important and must be done before the application update.)


3. Install the Application Update pak file.
   
   
   
   1. Ensure that you check both the boxes so that the “Reset out of the box content” option is selected. If you do not do this, some parts of the system may not be upgraded. This will not affect custom dashboards or other user-created content. Only the content that ships with the system that you ought not to have changed will be affected. If you do have customizations to the built-in content, you can clone them to preserve your changes.
   
   
   
   


4. Log in to the product UI as admin and ensure that your dashboards, adapter instances, and other data are present and working as expected.
   
   
   
   1. If it makes you go through the first run setup wizard again: don’t panic, its probably fine. Just choose evaluation mode and complete the wizard. Everything should be there when it’s complete. http://kb.vmware.com/kb/2132452
   
   
   2. If you end up with vCenter adapter instances that cannot be configured, missing dashboards or licenses, or end up loosing historical data, you should revert your snapshot and tell VMware about it immediately. If you don’t have a support contract, then make some noise on the communities site. VMware pays attention to this stuff and wants to prevent it from happening.
   
   
   
   


5. Once you’re sure that everything is running OK, delete the snapshots.

Conclusion

Although the upgrade process is a bit of an ordeal that can consume a good chunk of time, the features, stability, and performance of a successful upgrade really are worth it. VMware is working hard to make it better, and does listen to the things that are said on the communities site should you have any issues.

Read More

Horizon Workspace 1.8.1 Client Update & Heartbleed bug

Last night VMware quietly released a patched Windows and OSX Client for Horizon Workspace 1.5 and 1.8 bringing the client to version 1.8.1. This client release includes an updated OpenSSL 1.0.1g. Alongside the release is a KB with the remediation steps for the Client regarding the pesky CVE-2014-0160 vulnerability.

You can continue to track the VMware advisory for this vulnerability here: http://www.vmware.com/security/advisories/VMSA-2014-0004.html

Get the downloads from the VMware Product Page.

Read More