VMware EVO SDDC 1.2 Release Notes

VMware EVO SDDC | 1 SEP 2016 | Build 4271912

Note: As of September 1, 2016, VMware Cloud Foundation™ replaces EVO SDDC.
In this EVO SDDC 1.2 release, the software, user interfaces, and documentation refer to the prior EVO SDDC name. These references will be phased out in subsequent releases.

See the VMware Cloud Foundation product page for more details about the VMware Cloud Foundation product.

Release notes last updated: 11 DEC 2016
Check back for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

What's New

Look for New and Updated in the lists of known issues.

See important information about NSX 6.2.3a.

Among other features and enhancements, the VMware EVO SDDC 1.2 release includes the following key features:

  • Support for more hardware configurations
  • Support for an all-flash hardware configuration and the all-flash related Virtual SAN efficiency features: deduplication, compression, RAID 5 and RAID 6 erasure coding
  • Ability to increase capacity by adding additional servers and racks into the system
  • Automated configuration of network uplinks
  • Updated versions of VMware software

EVO SDDC 1.2 contains other VMware software. For information about what is new in those products, as well as their known issues and resolved issues, see the release notes for those software versions. You can locate their release notes from their documentation landing pages. See Documentation for the list.

VMware Software Versions and Build Numbers

Depending on the life cycle management (LCM) updates applied to your system, you might have different build numbers in your environment.

When an EVO SDDC 1.2 system is initially deployed, the system enables automated deployment of the following VMware software builds:

  • VMware EVO SDDC Manager 1.2 | 1 SEP 2016 | Build 4271912
  • VMware Platform Services Controller 6.0 Update 2 | 15 MAR 2016 | Build 3634791
  • VMware vCenter Server 6.0 Update 2 | 15 MAR 2016 | Build 3634791
  • VMware vSphere (ESXi) 6.0 Update 2 | 15 MAR 2016 | Build 3620759
  • VMware Virtual SAN 6.2 (shipping with ESXi 6.0 Update 2) | 15 MAR 2016 | Build 3620759
  • VMware NSX for vSphere 6.2.3a | 26 JUL 2016 | Build 4167369
  • VMware vRealize Operations 6.2.1 | 25 APR 2016 | Build 3774215
  • VMware vRealize Log Insight 3.3.1 | 14 MAR 2016 | Build 3644329
  • VMware vRealize Log Insight Agent 3.3.1 | 17 MAR 2016 | Build 3669972
  • VMware NSX content pack 3.3 for vRealize Log Insight | 18 APR 2016
  • VMware Tools 10.0.9 | 09 JUN 2016 | Build 3917699
  • VMware Horizon 6 version 6.2 | 08 SEP 2015 | Build 3005627
  • View management pack 6.2.1 for vRealize Operations | 11 MAY 2016 | Build 3598105
  • View content pack 1.0 for vRealize Log Insight
  • VMware App Volumes 2.10 | 24 NOV 2015

Important Information about NSX for vSphere 6.2.3a

Note: VMware identified an issue in VMware NSX for vSphere 6.2.3a that affects customers that deploy Distributed Logical Routers (DLRs). VMware is actively working towards releasing an updated 6.2.4 release that solves this problem. The update will be delivered to your Cloud Foundation system using the SDDC Manager life cycle management feature. For the current issue, refer to the workaround in:

VMware Software Edition License Information

The EVO SDDC Manager software is licensed under the EVO SDDC license. As part of this product, the EVO SDDC Manager software deploys specific VMware software products.

The following VMware software deployed by EVO SDDC Manager is licensed under the EVO SDDC license:

  • VMware vSphere
  • VMware Virtual SAN
  • VMware NSX

The following VMware software deployed by EVO SDDC Manager is licensed separately:

  • VMware vCenter Server
  • VMware vRealize Log Insight
  • VMware vRealize Operations
  • Content packs for Log Insight
  • Management packs for vRealize Operations
  • VMware Horizon 6
  • VMware App Volumes

For more information about the specific VMware software editions that are licensed under the licenses you have purchased, see the information available at the Cloud Foundation product page.

Documentation

To access the EVO SDDC 1.2 documentation, go to the VMware EVO SDDC documentation landing page.

To access the documentation for the included VMware software products, see their documentation landing pages and use the drop-down menus on the page to choose the appropriate version:

Browser Compatibility and Screen Resolutions for the EVO SDDC Web User Interfaces

The following Web browsers can be used to view the EVO SDDC Web user interfaces:

  • Mozilla Firefox: Basic Version 39 and Stabled Version 43.0.4
  • Google Chrome: Basic Version 42 and Stabled Versions 47.0.2526.70 (iOS) and 47.0.2526.106 (Windows, Linux)
  • Internet Explorer: 11.0.25 for Windows systems, with all security updates installed
  • Safari: Basic Version 9.0.3 on Mac only

For the Web-based user interfaces, the supported standard resolution is 1024 by 768 pixels. For best results, use a screen resolution within these tested resolutions:

  • 1024 by 768 pixels (standard)
  • 1366 by 768 pixels
  • 1280 by 1024 pixels
  • 1680 by 1050 pixels

Resolutions below 1024 by 768, such as 640 by 960 or 480 by 800, are not supported.

Known Limitations

  • You can access in-product context-sensitive help only if you have access to the Internet. If you do not have Internet connectivity, download PDF versions of the EVO SDDC documents from the VMware EVO SDDC documentation landing page.
  • For the locations in the user interface in which you enter a VLAN ID, a range of 3 to 3299 is enforced, even if the switches in your physical rack support a wider range of VLAN IDs. The 3 to 3299 range for VLAN IDs is chosen as the default supported range because it is a subset that is commonly supported across various switch manufacturers. The locations in the user interface in which you typically enter VLAN IDs are the system bring-up screens and the Virtual Infrastructure (VI) and Virtual Desktop Infrastructure (VDI) workload domain creation wizards.
  • Snapshots are not taken automatically for the virtual machines that are deployed during the bring-up process. After the bring-up process is completed, you can log in to the management domain's vCenter Server on the primary rack and take a manual snapshot of each deployed virtual machine.
  • When performing any workload-domain-related action — such as creating, expanding, or deleting a workload domain — wait until that workflow is completed 100% before initiating any other workload-domain-related action. Otherwise, unexpected results might occur. Existing workloads will continue to work fine on a workload domain undergoing expansion. It is recommended to avoid performing new management activity on a workload domain undergoing expansion. To check a workflow’s completion status, use the Workflows area of the System Status page in the EVO SDDC Manager user interface.

Known Issues

The known issues are grouped as follows:

Bring-Up Known Issues

  • If the time sync process fails on a host but POSV passes with no issues, you are not prevented from continuing the system bring-up process even though tasks later on in the bring-up process might fail
    If the time sync process indicates failure on a host, you can continue in the UI to the POSV screen and run the POSV process to help identify issues on the host that might have caused the failure. However, due to this issue, if the POSV process subsequently passes, the Continue button is available and allows you to proceed in the bring-up process even though the time was not synchronized on that host. If you click Continue to proceed instead of Retry to rerun the time sync process, bring-up tasks that are performed later might fail, such as setting the NTP or deploying the PSC and vCenter Server appliances.

    Workaround: To avoid any unexpected issues, if the time sync process indicates a host has failed but the POSV process passes, click Retry after the POSV is done to ensure the time synchronization is rerun.

  • Alerts raised during POSV do not contain a rack name in their description
    Because the rack name is not specified by the user until after the IP allocation step in the system configuration wizard, a rack name is not available to display in alerts raised prior to that step. Alerts that are raised subsequent to that step do contain the user-specified rack name.

    Workaround: None.

  • POSV might take more than an hour to report success
    The POSV process occurs as one of the first steps in the system bring-up process. At the start of the POSV process, the Physical Resources Manager (PRM) performs inventory collection, and the POSV process waits until this inventory collection is completed. If the inventory collection takes up to an hour, the overall POSV process can take more than hour to complete.

    Workaround: None.

  • System bring-up process might fail at task VC: Apply vCenter License
    Due to intermittent timing issues, sometimes the system bring-up process might fail at the VC: Apply vCenter License task with the following exception
    java.lang.RuntimeException: Can't create VsphereClient instance

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

  • System bring-up process might fail at task Network: Assign Public Management IP address to Switches
    If intermittent connectivity to the switches occurs during the bring-up process, the process might fail at the Network: Assign Public Management IP address to Switches task. The vrm log file will have error messages stating the updates of the default route configuration on the switches did not receive responses from the HMS Aggregator, and the HMS log will have lines showing response code 500 for the PUT API request making the route update on the switches.

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

  • New The system bring-up process might fail at task NSX: Register vCenter with error NSX did not power on on time
    The bring-up process fails because the NSX Controller virtual machines did not power on during the wait time set in the NSX: Register vCenter task.

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

  • System bring-up process might fail at task ESX: Create NFS Datastore
    If intermittent connectivity to an ESXi host occurs during the bring-up process, the process might fail at the ESX: Create NFS Datastore task with the following exception
    Exception trying to create NAS DS on host host-ip-address
    

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

  • System bring-up process might fail at task LCM: Configure LCM Repo VM
    If intermittent connectivity to the VMs in the management cluster occurs during the bring-up process, the process might fail at the LCM: Configure LCM Repo VM task with the following exception
    SSH: Failed to establish SSH session

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

Multi-Rack Bring-Up Known Issues

  • If the first rack’s ESXi host N1 is down when you start the bring-up process on an added rack, the starting Time Sync user interface screen for the new rack’s bring-up process appears blank and no progress is visible
    At the start of the bring-up process on the new rack, the VRM virtual machine on the new rack tries to connect to the N1 host in the first rack to obtain the host's time. The system uses the time of a host in the first rack for the time sync process on the new rack to ensure that time is synchronized. If the first rack's N1 host is down, the system continues to try to connect to that host for more than 10 minutes before attempting to obtain the time from another host in the first rack. During this retry time, the Time Sync screen appears blank, the bring-up process is delayed, and you cannot tell if progress is being made.

    Workaround: If the user interface appears blank when you start the bring-up process on an additional rack, first wait at least 15 minutes and then reopen to the added rack's bring-up starting URL https://192.168.100.40:8443/vrm-ui. If the screen shows the time set successfully for the listed components, then you can proceed as usual.

  • The system bring-up user interface does not prevent you from starting the process on an additional rack even if you did not perform all of the required manual steps for rack addition, such as copy the file encryption keys and syncing properties, resulting in failure of system bring-up on the additional rack
    As documented in the EVO SDDC Overview and Bring-Up Guide, before you run the system bring-up process on an additional rack, you must first bootstrap EVO SDDC Manager on the new rack from the primary rack, and then successfully perform some manual steps such as copying the file encryption keys (vrmsecurity.keystore) from the primary rack to the additional rack and syncing properties between the racks. These required steps are documented in the Manual Steps for Rack Addition procedure in the guide. If you do not appropriately perform that procedure and you begin using the system bring-up wizard directly from the EVO SDDC Manager on the new rack, the system’s data fails to properly initialize and system bring-up on the new rack fails. At this point, you must manually clean-up the data about the existing workflow.

    Workaround: Before triggering the system bring-up wizard on the new rack, ensure you complete the procedures as documented in the EVO SDDC Overview and Bring-Up Guide.

  • Tasks deploying Platform Services Controller (PSC) should not be displayed during the system bring-up process on racks added after the first rack
    While running the system bring-up process on a second rack, the following tasks are displayed
    SUCCESSFUL - ESX: Set VMs Startup Order After 2nd PSC Deployed
    SUCCESSFUL - PSC: Deploy 2nd PSC
    SUCCESSFUL - ESX: Set VMs Startup Order After 1st PSC Deployed
    SUCCESSFUL - PSC: Deploy 1st PSC
    Because the bring-up process on additional racks uses the PSC on the first rack, these tasks should not be displayed.

    Workaround: Ignore these tasks.

  • The system bring-up process on an additional rack might fail at task VC: Enable Virtual SAN with error Cannot enable force provision
    The EVO SDDC bring-up process might fail with following exception
    com.vmware.vrack.vrm.workflow.tasks.TaskException: Cannot enable force provision.
    This might happen because communication between Virtual SAN and SMS (Storage Management Service) is broken. SPBM (Storage Profile-Based Management, also known as Storage Policies) does not work in this situation.

    Workaround: Follow the procedure below.

    1. Using the vSphere Web Client, navigate to the domain.
    2. Select Hosts and Cluster.
    3. Select vCenter and then click Manage.
    4. Click the Storage Providers tab.
    5. If the status of storage providers is disconnected, click Resync.
    6. Click Enable VSAN.
  • The system bring-up process on an additional rack might fail at task NSX: Register vCenter with error NSX did not power on on time
    The bring-up process fails because the NSX Controller virtual machines did not power on during the wait time set in the NSX: Register vCenter task.

    Workaround: In the bring-up user interface, click the Retry button to perform the task and proceed with the bring-up process.

  • Even though the bring-up user interface displays 100% completion for an added rack’s bring-up process, there might be a delay of 30 minutes or longer before you can rotate passwords or interact with the newly added rack
    This delay occurs during the time the system is setting up the IPSec tunnels in the background. IPSec is a protocol that secures communication over IP networks. While the IPSec tunnels are being set up, interactions with the added rack are blocked.

    Workaround: To determine when the IPSec tunnels are ready and you can interact with the newly added rack, you can check the IPSec tunnel status from the added rack's VRM virtual machine. When all of the tunnels report ESTABLISHED status, you can rotate passwords and interact with the newly added rack.

    To determine that the IPSec tunnels are ready:

    1. Using a method for remote secure connections such as SSH, log in to the added rack's VRM virtual machine using the root account and its password. The root account password is provided along with the bootstrap password from your hardware provider. By default, the root account password is the same as the bootstrap password.
    2. In the VRM virtual machine, change directories to /home/vrack/vrm/ism/scripts directory.
    3. Check the IPSec tunnel status by running the command:
      ipsecutil.py -c tunnels-up
      
      Sample output:
      VRM2toISVM2Tunnel,VRM2toISVM3Tunnel,VRM2toISVM1Tunnel
      ESTABLISHED
      VRM2toISVM2Tunnel:ESTABLISHED
      ESTABLISHED
      VRM2toISVM3Tunnel:ESTABLISHED
      ESTABLISHED
      VRM2toISVM1Tunnel:ESTABLISHED
      
      Line 1 is the list of this VRM VM's defined IPSec tunnels.
      Line 2 after the list of IPSec tunnels reports the overall status.
      The subsequent lines report the status of each defined tunnel. If any of the tunnels are in a state other than ESTABLISHED, such as CONNECTING, the system is not yet ready.

    Note: During the tunnel set-up process, the system might remove some tunnel definitions and create new tunnel definitions. As a result, each time you run the command to query the status, you might see different and additional names listed in the output. Towards the end of the tunnel set-up process, the list stabilizes.

    When the command output indicates the IPSec tunnels are established, you can perform the rotate password procedure in the same VRM VM login session. See the EVO SDDC Overview and Bring-Up Guide for the Change Passwords of Rack Components procedure details.

Post Bring-Up Known Issues

  • The system's bring-up process creates a distributed port group vRack-DPortGroup-VXLAN in the management domain that does not get used in the steady-state system
    When you look at the distributed port groups that are created for the management domain, you see a distributed port group named vRack-DPortGroup-VXLAN, and the management domain's ESXi hosts have access to it. However, that distributed port group is not used by the system. The VXLAN distributed port group that the system uses is the one that is automatically created by the NSX Manager during its VXLAN preparation process. The one that is used is named vxw-vmknivPg-dvs-nnnn, where nnn is a system-generated identifier.

    Workaround: None. Ignore the distributed port group named vRack-DPortGroup-VXLAN.

  • When you log in to the vSphere Web Client using the superuser SSO account defined during bring-up, you do not see all of the NSX Manager features in the Networking & Security area
    During the bring-up process, in the Create a Superuser Account screen, you entered a user name and password. Using that input, the system created a superuser account as your main login account for your system. Due to this issue, the system does not automatically assign the NSX Manager Enterprise Administrator role to that superuser account. The NSX Manager Enterprise Administrator role grants permissions for NSX operations and security to a vCenter Server user.

    Workaround:

    1. Log in to the EVO SDDC Manager Web interface using the system-managed SSO administrator account credentials, and then open the vSphere Web Client using the vCenter launch link on the management domain’s details page.
    2. In the vSphere Web Client, navigate to the Networking & Security > NSX Managers page, click the listed NSX Manager, click Manage, and click Users.
    3. Click the green + icon and use the Assign Role dialog to assign the Enterprise Administrator role to your system’s superuser account.

    After granting the NSX Manager Enterprise Administrator role to the superuser account, you can now log in to the vSphere Web Client with that account and the Networking & Security features are available to that user.

    For more details about NSX Manager roles, see these topics in the NSX 6 Documentation Center:

  • The lookup-password, rotate-all, and rotate-password-vrops commands might refer to the release notes for steps to set up account credentials for the vRealize Operations Manager virtual machine even though the steps are actually in the EVO SDDC Overview and Bring-Up Guide
    After running the system bring-up process on a physical rack, you change the passwords of the rack’s physical and logical components using the vrm-cli.sh script located in the /home/vrack/bin directory in the VRM virtual machine on that rack. This script has commands lookup-password for looking up the existing rack components’ passwords and rotate-all for changing the passwords. The steps for looking up and changing the passwords of the rack’s physical and logical components are documented in the EVO SDDC Overview and Bring-Up Guide. However, if the system cannot locate the setting for the vRealize Operations Manager virtual machine’s root password, the following message appears after running the lookup-password, rotate-all, and rotate-password-vrops commands (where nn.nn.nn.nn is the IP address of the vRealize Operations Manager VM):
    Password for vROps host at nn.nn.nn.nn is not set up properly.
    Please refer to release notes how to set up those credentials.

    Workaround: The steps for setting up the credentials are documented in the EVO SDDC Overview and Bring-Up Guide.

  • Host sometimes hangs after doing a normal reboot
    Due to a known Virtual SAN issue, in an EVO SDDC system that has Dell R630 or Dell x730d servers and with certain controllers, sometimes a host hangs after doing a normal reboot. For a complete list of affected controllers, see VMware Knowledge Base article 2144936.

    Workaround:

    1. Ensure the storage-controller drivers and firmware are updated on each host according to the information in the VMware Compatibility Guide for Virtual SAN for PERC H370:http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio&productid=34853&deviceCategory=vsanio&details=1&vsan_type=vsanio&keyword=h730
    2. Apply the settings as described in the VMware Knowledge Base article 2144936.

EVO SDDC Manager Known Issues

  • The Rack Details screen might display a warning with code VRM040901E and message "Error loading rack details" when backend services are restarted in the VRM virtual machine
    Some administrative operations, such as rotating passwords, require you to stop the vrm-watchdogserver and vrm-tcserver services in a rack’s VRM virtual machine and then restart those services after you have completed the administrative operation. During the restart process, the system gathers hardware inventory information, which can take up to 15 minutes. If you navigate to the Rack Details screen within 15 minutes of restarting the services, the warning message might appear because the system is still gathering the up-to-date hardware information.

    Workaround: After restarting those services in the VRM VM, wait at least 15 minutes before navigating to the Rack Details screen.

  • When a management domain is configured with three ESXi hosts and one of those ESXi hosts goes down, even though you are able to use the EVO SDDC Manager user interface to start the workload domain creation process, the creation fails it tries to create the VI environment's required VMs
    By default during the system's bring-up process, the system's management domains are configured to have three ESXi hosts. If you have not expanded a management domain to have more than three hosts, you might encounter this issue. If a management domain has only three ESXi hosts and one of the hosts goes down, the management VMs continue to run, and you can use the VI workload domain creation wizard to start the creation workflow. However, due to this issue, if the workflow is using the management domain that has only two running hosts, the workflow will fail when it gets to its tasks for deploying the VI environment's required VMs in that management domain, such as the vCenter Server VM and NSX Manager VM. The reason is because the management domain's cluster is configured by default with Virtual SAN setting HFT=1, and that setting requires a minimum of three running hosts to successfully create new VMs in that cluster.

    Workaround: If a workload domain creation fails due to this issue, expand the management domain that the workflow is using and then restart the failed workflow.

  • In the Workflows screens, for a workload domain deletion workflow, the "Destroy vms on hosts" subtask does not display the details of the VMs and ESXi hosts affected by that subtask.
    For a workflow, you can navigate to see the details of the workflow's subtasks from the System Status > Workflows screen in the EVO SDDC Manager Web interface, and when you expand a subtask, the screen usually displays details about the system artifacts that are being operated on by that subtask. For the workflow in which a workload domain is being deleted, the user interface does not display any details about the VMs or ESXi hosts on which the "Destroy vms on hosts" subtask is operating, such as the number of VMs which are deleted from which ESXi host.

    Workaround: None. You can examine the log files to determine the details of the VMs and ESXi hosts that are involved in the subtask.

  • Sometimes the VRM VM cannot detect the network, while all other VMs on the same ESXi host are unaffected.
    The VRM VM is the virtual machine that runs the EVO SDDC Manager application. Sometimes the VM's console displays the message "No Networking Detected" and indicates the VRM VM cannot detect the network. Only the VRM VM experiences this issue. This issue is due to an issue in the VM's virtual hardware settings, and the VM's network adapters cannot detect the network.

    Workarounds:

    • Reboot the VRM VM by shutting down the VRM VM's guest OS and then powering the VM back on. You can use the vSphere Web Client to access the management domain's vCenter Server, navigate to the VRM VM, and run the Power > Shut Down Guest OS operation on the VM, and then the Power > Power On operation. Rebooting the VM in this way might resolve the VM's connectivity.
    • If the "No Networking Detected" message still appears on the VRM VM's console, edit the VRM virtual machine's VM Hardware settings to delete the virtual network adapters and then add new ones and configure those new ones to the vSphere distributed port groups that the previous ones had used (vRack-DPortGroup-Mgmt and vRack-DPortGroup-NonRoutable).
  • An expansion workflow that involves adding more than one ESXi host to a management or workload domain is marked successful, even though when the hosts were added to the domain's vCenter Server cluster, the NSX Manager Host Preparation process failed to complete on one or more hosts
    During an expansion workflow, the hosts are added to the vCenter Server cluster that underlies the management or workload domain. When hosts are added to a vCenter Server cluster that has NSX enabled on the cluster, one of the tasks involves preparing the newly added hosts, as described in the Prepare Hosts on the Primary NSX Manager topic in the NSX 6.2 documentation. Part of this host preparation process involves a scan of each added ESXi host prior to installing the required NSX software on that host. If the scan on a particular host fails for some transient reason, the NSX Manager host preparation process fails for that host. However, this failure condition is not reported to the expansion workflow and the workflow appears as successful in the EVO SDDC Manager Web interface.

    Workaround: When performing an expansion workflow that involves multiple hosts, when the EVO SDDC Manager Web interface indicates the workflow has completed, perform the follow steps to verify the NSX host preparation was successful for each added host, and if not, resolve the issues reported by NSX.

    1. Using the vSphere Web Client, log in to the vCenter Server instance for the management or workload domain that was expanded.
    2. In the vSphere Web Client, examine the NSX Manager host preparation state by navigating to Networking & Security > Installation and clicking the Host Preparation tab.
    3. On the Host Preparation tab, expand the cluster if it is not already expanded, and examine the data reported for each host in the Installation Status column and VXLAN column:
      • If the Installation Status column reports green checkmarks and "Configured" in the VXLAN column for all hosts, the added hosts were successfully prepared.
      • If the Installation Status column displays "Not Ready" and the corresponding VXLAN column displays "Error" for a host, resolve the error by right-clicking on the VXLAN column's "Error" and clicking Resolve. This action also applies the VXLAN distributed switch port group to that host.
  • Because no unique identifier is used to identify an EVO SDDC system, when you deploy more than one EVO SDDC system in your environment, you cannot use the same level of Active Directory (AD) integration for both systems
    For the first EVO SDDC system deployed in your environment, you would configure AD authentication by adding your AD as an identity source to the Platform Security Controller instances using the Active Directory (Integrated Windows Authentication) option and joining the vCenter Single Sign-On server to the AD domain. Due to this issue, for additional systems deployed in your environment, you cannot do that same configuration.

    Workaround: For additional systems, the Active Directory as an LDAP Server option can be used to add your AD as an identity source to the Platform Security Controller instances in those systems.

  • In the Data Center Connections screen, the system does not prevent you from associating a VDI workload domain's data center network with management and VI workload domains while creation of the VDI workload domain is still in progress or has failed
    One of the actions you perform in the Data Center Connections screen is associating defined data center network configurations (the subnet ranges) with management and workload domains. Normally the system prevents you from associating a data center network that is already associated with a VDI workload domain with any other management or workload domain. However, due to this issue, if the VDI workload domain creation is still in progress, or has failed, the system does not prevent you from using the Data Center Connections screen to associate that VDI workload domains' data center network with management or VI workload domains. If the VDI workload domain creation process has reached the tasks for deploying the VI workload domain that underlies the VDI workload domain, the Data Center Connections screen will show the data center network configuration associated with that VI workload domain, until the VDI workload domain creation process is finished.

    Workaround: To avoid this issue, refrain from performing the association operation using the Data Center Connections screen while a workload domain creation is in progress.

  • Navigating through a wizard using the top navigation bar does not save the values you specify in the wizard screens, causing errors that result in the workflow to fail
    Due to this issue, when you use the top navigation bar in workflow wizards to move between screens instead of using the Next and Back buttons, the values you enter and selections you make in the screens might not be saved to the system. As a result, the workflow fails.

    Workaround: Always use the Next and Back button to move through wizard screens.

  • A workload domain’s workflow can fail if a VM in the management domain on which the workflow depends is in non-operational state
    Workflows to deploy, delete, and expand workload domains can fail if some of the management domain’s virtual machines are in an invalid state, down, or temporarily inaccessible. The EVO SDDC Manager does not prevent you from initiating and submitting a workflow when one of the VMs is in an invalid state. These virtual machines include the PSC VMs, vCenter Server VMs, Infrastructure Services Manager VMs, vRealize Operations Manager VM, Log Insight VM, NSX Manager VM, and, in a dual-rack system, the VRM VM. If you submit the workflow and one of those virtual machines becomes temporarily inaccessible as the workflow is performed, the workflow will fail.

    Workaround: Before initiating a workflow, review the state of the management domain’s virtual machines to see that they are all in a valid (green) state. You can see the virtual machines by launching the vSphere Web Client from the domain details of the management domain.

  • In a multi-rack system, when a rack’s VRM virtual machines is down, the Physical Resources screen displays empty values of CPU, Memory, and Storage for that rack
    Each physical rack has a VRM VM running on a host in that rack. If a rack’s VRM VM goes down, its CPU, memory, and storage values are not retrievable and the Physical Resources screen displays empty values for that rack.

    Workaround: Use the System Status screen to examine the system alerts and determine the appropriate steps to bring the rack’s VRM VM back online.

  • After powering off a switch, it can take up to 5 minutes for the switch’s powered off state to be reflected by the UI, then the UI might display error code VRM04091E
    In a switch’s Switch Details screen, the Powered field indicates the power state of the switch: on or off. When you manually power off a switch, the Switch Details screen continues to indicate the switch is powered on for up to 5 minutes. However, the appropriate system events and alerts are raised. If you had manually powered down a spine switch, at 5 minutes after you powered off the switch, when you try to view the Rack Details screen, the user interface might display an error message "Error loading rack details. Please contact VMware support with the code: VRM040901E."

    Workaround: None. You can use the System Status screen to see the events and alerts about the switch’s current operational status.

  • When using the EVO SDDC Manager Web interface’s Uplink screen to update L3 connectivity settings, the Uplink screen does not indicate which of the ToR switches has the L3 uplink configured on it
    When an uplink is configured to L3 mode, only one of the two ToR switches has an uplink port. The EVO SDDC Manager UI does not indicate which ToR switch is connected to the upstream router.

    Workaround: When you use the Uplink screen to change uplink connectivity settings, perform the following steps.
    Note: Changing the settings triggers uplink reconfiguration on the switches. Because the reconfiguration process might take a few minutes to complete, connectivity to the corporate network might be lost during the process. To avoid losing connectivity with the EVO SDDC Manager, it is strongly recommended that you are connected to port 48 on the management switch when updating the settings using the Uplink screen.

    1. Connect to port 48 on the management switch and log in to the EVO SDDC Manager Web interface using that connection.
    2. On the Uplink screen, configure the L3 uplink and click SAVE EDITS.
    3. Re-configure your upstream router to use the new network settings that you specified in step 2.
    4. Wait at least 3 minutes.
    5. Try connecting the upstream router to the top ToR switch.
    6. Test the new uplink connectivity by disconnecting from port 48 on the management switch and connecting to the virtual rack with the new uplink configuration.
    7. If you are unable to reconnect to the virtual rack, try connecting the upstream router to the bottom TOR switch.
    8. If you are unable to connect to the virtual rack, reconnect using port 48 on the management switch and try reconfiguring your network to the original configuration.
    9. If you cannot connect to the virtual rack with either configuration, contact VMware Support.
  • Existing trunk ports on ToR Cisco Nexus 9K switches are assigned to new VLANs when a VI or VDI workload domain is created
    Due to this issue, when new VLANs are entered during creation of a VI or VDI workload domain, the external VLAN and other VLANs created specifically for the new workload domain get assigned to all existing trunk ports on the ToR Cisco Nexus 9K switches, including the uplink and management cluster bonds.

    Workaround: None

  • When a new VI or VDI workload domain is created, the host-specific ports configured on the ToR Cisco Nexus 9K switches become members of all existing VLANs
    When the host-specific ports are configured on the ToR switches for a new VI or VDI workload domain, the ports are put into VLAN trunk mode using command switchport mode trunk. Due to this issue, the ports result in being members of all existing VLANs.

    Workaround: None

  • In the Switch Details screen, status of a switch port that is used to connect a commissioned server will be shown as down even after successful commissioning.
    If a previously unused switch port is used to connect a new server that is commissioned into the rack, the port status, which is expressed in terms of color in the Switch Details screen, does not change. The color of the port depicted in the Switch Details screen remains the same color as an unused port (yellow), even if the port state goes down or comes back up.

    Workaround: None. For monitoring the health of the server’s connection to that switch port, because the server side NIC port will show as down if the link is faulty, you can use any alerts on the commissioned server’s NIC port as an indication of failure of the switch port.

  • Log files might contain occurrences of "IaaS"
    The phrase "IaaS" might appear in the system's log files instead of "Virtual Infrastructure" or "VI".

    Workaround: None. If you see occurrences of "IaaS" in the log files, they are referring to the system's Virtual Infrastructure (VI) features.

Virtual Infrastructure Workload Domain Known Issues

  • When deleting a failed VI workload domain, the deletion workflow fails at task "Restore hosts from backup"
    If the VI workload domain creation process fails before successfully completing the task of taking a backup of a host's initial state, when you subsequently start the process to delete the failed workload domain, the deletion workflow fails at the task "Restore hosts from backup". This failure occurs when the system cannot find an ESXi host's backup file in the /store/backup folder on the host due to the workload domain creation process failing before the system could write a backup file to that location.

    Workaround: For each of the ESXi hosts that are assigned to the VI workload domain you are trying to delete, manually backup the host and place the backup file in the host's /store/backup folder. Then restart the workload domain deletion workflow.

  • In the VI workload domain creation and expansion wizard screens, an error occurs if you specify a storage value equal to the displayed available total
    Both the VI workload domain and expansion wizard screens display a value for the total available storage next to the Storage entry field. However, if you specify that displayed total available value for the desired storage, an error is displayed and you cannot complete the wizard to start the workflow.

    Workaround: Instead of specifying the total available value in the Storage field, you can enter that value minus 0.01 (total-available – 0.01). For example, if the screen displays 7.66 as the available storage, use 7.65.

  • The VI workload domain creation and expansion workflows might fail at task "ConfigureVCenterForLogInsightTask" due to a failure to connect to the system's integrated vRealize Log Insight instance
    During the VI workload domain creation and expansion workflows, if the system cannot connect to the integrated vRealize Log Insight instance, the workflow fails at the "ConfigureVCenterForLogInsightTask" task and you see an exception in the log with a 500 HTTP error code:
    [com.vmware.vrack.vrm.workflow.tasks.loginsight.ConfigureVCenterForLogInsightTask] Exception while doing the integration: Create session to LogInsight Failed : HTTP error code : 500

    Workaround: Restart the vRealize Log Insight's virtual machine by using the management domain's vCenter Server launch link to open the vSphere Web Client and using the vSphere Web Client user interface to restart the vRealize Log Insight's virtual machine. Then restart the failed workflow.

  • The VI workload domain creation workflow might fail at task "VC: Deploy vCenter" due to a failure to connect to the system's Platform Services Controller instances
    During the VI workload domain creation workflow, if the system cannot connect to the integrated Platform Services Controller instances, the workflow fails at the "VC: Deploy vCenter" task and you see errors in the log such as:
    Unexpected error while verifying Single Sign-On credentials: [Errno 111]
    Connection refused
    Cannot get a security token with the specified vCenter Single Sign-On configuration.

    Workaround: Restart the system's PSC-2 virtual appliance, then the PSC-1 virtual appliance, then the vCenter Server virtual appliance. Wait until each virtual appliance is up and running before restarting the next one. Then restart the failed workflow.

  • In each of the VI workload domain creation wizard's networking user interface screens, the final value entered in the Subnet field must be zero (0)
    In the Subnet fields of the VI workload domain creation wizard's networking screens, a zero (0) must be entered in the final position, such as nn.nn.nn.0. If anything other than a zero is entered for the final value, a validation error occurs after you enter values in the Subnet Mask field.

    Workaround: None.

  • In a multirack system, a VI workload domain creation workflow might fail at task "NSX: Register vCenter" with an error stating NSX did not power on
    The creation workflow fails because the NSX Controller virtual machines did not power on during the wait time set in the NSX: Register vCenter task. The error message is NSX didn't power on on time.

    Workaround: Restart the failed workflow using the Restart Workflow action in the workflow’s status screen.

  • On the Review page of the VI workload domain creation wizard, the Download and Print buttons are not operational
    Due to this issue, when you reach the Review step of the VI workload domain creation wizard, you cannot use the Download or Print buttons to create a printable file of the displayed information for future reference.

    Workaround: None. At the Review step of the wizard, you must manually capture the information for future reference, for example by taking screen captures of the displayed information.

VDI Workload Domain

  • The VDI workload domain creation workflow might fail at task "Instantiate Horizon View Adapter"
    Due to intermittent timing issues, the VDI workload domain creation workflow sometimes fails at the Instantiate Horizon View Adapter task with the following exception error in the log:

    com.vmware.vrack.vdi.deployment.tools.tasks.VDIWorkflowException: "Unable to create vROps REST client"
    

    As a result, the pairing credential between the vRealize Operations Manager instance and the VDI environment is in a partially instantiated state and must be deleted before restarting the workflow.

    Workaround: Manually delete the pairing credential that is associated with the workload domain's Horizon Connection server and then restart the failed workflow using the Restart Workflow action in the workflow’s status screen using these steps:

    1. Verify that you have the IP address for the first Horizon Connection server that was deployed for this VDI workload domain, such as 10.11.39.51 You will use that IP address to identify which pairing credential to delete.
    2. Log in to the vRealize Operations Manager Web interface. You can use the launch link in the management domain's details screen to open the log in screen.
    3. From the vRealize Operations Manager Web interface's Home screen, navigate to the Credentials screen by clicking Administration > Credentials.
    4. Locate the pairing credential having a name in the form of vdi-view-adapter-IPaddress, where the IP address matches the one you obtained in step 1. For example, if the Horizon Connection server has IP address 10.11.39.51, the displayed pairing credential name is vdi-view-adapter-10.11.39.51.
    5. Select that pairing credential and delete it.
    6. In the workflow's status screen, restart the failed workflow using the Restart Workflow action.
  • When creating a VDI workload domain with specified settings that results in the system deploying two vCenter Server instances, the creation workflow might fail at the "ESXI: Incremental LI Integration" task
    Depending on the value for the Max Desktops [per vCenter Server] setting in the VDI Infrastructure screen and your choice for the number of desktops in the VDI workload domain creation wizard, the system might need to deploy more than one vCenter Server instance to support the desired VDI workload domain. As part of this deployment, the system starts two VI workload domain creation workflows. One of the VI workload domain creation workflows might fail at the task "ESXI: Incremental LI Integration" with an error message about failure to connect ESXi hosts to vRealize Log Insight:
    hosts to LogInsight failed : HTTP error code : 404 : Response :

    Workaround: Use the Physical Resources screens to verify that the ESXi hosts that the failed workflow is trying to use are all up and running. Use the vSphere Web Client to verify that the vRealize Log Insight VMs in the system are all up and running. Ensure that the ESXi hosts involved in the failed workflow and the vRealize Log Insight VMs are in a healthy state, and then click Retry in the failed workflow.

  • In a multirack system, a VDI workload domain creation workflow might fail at task "NSX: Register vCenter" with an error stating NSX did not power on
    The creation workflow fails because the NSX Controller virtual machines did not power on during the wait time set in the "NSX: Register vCenter" task. The error message is NSX didn't power on on time.

    Workaround: Restart the failed workflow using the Restart Workflow action in the workflow’s status screen.

  • In the Status - Workflows screens of the EVO SDDC Manager user interface, for some VDI workload domain expansion workflows, you might see some tasks listed in new state even though the overall workflows are completed and successful
    When a VDI workload domain is expanded more than once and the latest expansion workflow resulted in increasing the number of hosts or adding a vCenter Server instance to the workload domain, some tasks related to the workload domain's latest expansion workflow are added to the displayed subtasks list of the previously completed expansion workflows. The added tasks are listed as having NEW state. The added tasks are:

    • AddExternalNetworksToVcentersTasks
    • Create vCenter clusters taskID
    • SuppressExpirationTask

    Where taskID is a system-generated integer, such as 1, 2, and so on.

    Workaround: None. Ignore these displayed tasks.

  • During the second expansion of a VDI workload domain, partitioning of the Virtual SAN cluster can occur.
    Partitioning of the Virtual SAN cluster might occur in this scenario:

    1. You create a VDI workload domain.
    2. You expand the VDI workload domain with settings such that the system deploys an additional vCenter Server instance. The expansion workflow deploys an additional vCenter Server instance when the number of desktops specified in the expansion wizard is greater than the Max Desktops [per vCenter Server] setting on the Settings > Physical Rack Settings > VDI Settings screen. The default setting is 2000. When this expansion is completed, the VDI workload domain has two clusters.
    3. You expand that VDI workload domain again, to add more desktops such that the expansion workflow adds hosts to each cluster. The Virtual SAN cluster partitioning occurs during this expansion workflow.

    Workaround: When expanding a VDI workload domain, follow this guidance:

    • In the expansion wizard, do not specify a number of desktops larger than the default Max Desktops [per vCenter Server] setting of 2000.
    • Use the VDI workload domain expansion workflow to add more desktops as long as the total number of desktops in the workload domain will remain within the Max Desktops [per vCenter Server] setting.
    • The data center connection subnet used by the VDI workload domain must be large enough to support more than 1000 desktops. If the subnet is not large enough prior to expansion, first associate a new data center connection with the VDI workload domain that can support more than 1000 desktops. Ensure that the data center network meets the communication and routing requirements as documented in the Administering VMware EVO SDDC guide.

Life Cycle Management (LCM) Known Issues

  • LCM update logs saved in two folders
    LCM update logs are being saved in two folders:

    • /home/vrack/lcm/upgrade
    • .[/home/vrack/lcm/upgrade]

    Workaround: Review logs in both folders.

  • ESXi upgrade fails with message Cluster not in good condition
    Virtual SAN clustering on the host was not enabled resulting in the upgrade failure.

    Workaround:

    1. Open the vCenter Web Client.
    2. Navigate to Hosts and clusters >ClusterName >Monitor >Issues.
    3. Fix the Virtual SAN issue.
    4. Reschedule the update.
  • ESXi and vCenter update on a host might fail in the task of exiting maintenance mode
    Sometimes during an ESXi and vCenter update process, a host might fail to exit maintenance mode, which results in setting the status of the update process to failed. During an update, the system puts a host into maintenance mode to perform the update on that host, and then tells the host to exit maintenance mode after its update is completed. At that point in time, an issue on the host might prevent it from exiting maintenance mode.

    Workaround: Attempt to exit the host from maintenance mode using the vSphere Web Client: locate the host, right-click it, and click Maintenance Mode > Exit Maintenance Mode. Address any issues reported about that host in the vSphere Web Client. When you see the host has successfully exited maintenance mode, return to the EVO SDDC Manager user interface and retry the update when it appears in the Available Updates list.

Monitoring Known Issues

  • When a Cisco ToR switch is powered down or up, the TOR_SWITCH_DOWN event is not listed in the System Status - Events screen in the EVO SDDC Manager Web interface, even though the event is visible in the integrated vRealize Log Insight instance
    For systems that have Cisco ToR switches, when the switch is powered down, the TOR_SWITCH_DOWN event is not being written to the software location that populates the System Status - Events screen, even though the event is sent as expected to the vRealize Log Insight integrated in the system. As a result, even though you can see the reported events using the vRealize Log Insight Web interface, the TOR_SWITCH_DOWN events from Cisco ToR switches will not appear in the EVO SDDC Manager's Events screen.

    Workaround: Use the integrated vRealize Log Insight Web interface as the reporting source for any TOR_SWITCH_DOWN events.

  • When a Quanta server is powered down (off), the CPU_CAT_ERROR (CPU Catastrophic Error) event is generated
    When a Quanta server is powered down, the SERVER_DOWN event is generated, which is the expected behavior. The issue is the CPU_CAT_ERROR event is also generated when a Quanta server is powered down.

    Workaround: None

Hardware and FRU Replacement Known Issues

  • Commissioned host has 0 value for CPU, Memory, and Storage on EVO SDDC Manager Dashboard
    If you specify incorrect BMC credentials in the server-commission.properties file, the commissioned host will have 0 CPU, Memory, and Storage. The Host Details page displays the message Host should be powered on to view details.

    Workaround:

    1. Decommission the host.
    2. Update the /home/vrack/VMware/vRack/server-commission.properties file with correct values for BMC user name and password.
    3. Commission the host.
  • Server decommissioning workflow might fail at task Enter hosts maintenance mode when a server being decommissioned has many VMs running on it
    This issue might occur when the servers are part of a workload domain in which many VMs are running on the servers, such as a VDI workload domain with many virtual desktops. When a server is decommissioned, the server’s ESXi host is put into maintenance mode so that it can be removed from the workload domain’s vCenter Server cluster. Prior to an ESXi host entering maintenance mode, the VMs running on the host are migrated to another ESXi host in the workload domain using vMotion. The system waits until all ESXi hosts participating in the decommissioning workflow have entered maintenance mode before beginning the next workflow task. If there are many VMs running on one of the hosts, the migration process can take longer than for the others. After a number of attempts to verify that the host has not entered into maintenance mode, the system flags the workflow task as failed.

    Workaround: Restart the failed workflow using the Restart Workflow action in the workflow’s status screen.