VMware ESXi 6.0 Update 3 Release Notes
Updated on: 14 MARCH 2017
ESXi 6.0 Update 3 | 24 FEB 2017 | ISO Build 5050593
Check for additions and updates to these release notes.
What's in the Release Notes
The release notes cover the following topics:
Updated ESXi Host Client: VMware ESXi 6.0 Update 3 includes an updated version of the ESXi Host Client,
version 1.14.0. The updated Host Client
includes bug fixes and brings it much closer to the functionality provided by the vSphere Client. If you
updated the Host Client through ESXi 6.0 patch releases, then install version 1.14.0 provided with ESXi 6.0
Update 3. In addition, new versions of the Host Client continue to be released through the
VMware Labs Flings Web site.
However, these Fling releases are not officially supported and not recommended for production environments.
Support for TLS: Support for TLSv1.0, TLSv1.1 and TLSv1.2 are enabled by default and configurable
for ESXi 6.0 Update 3. Learn how to configure TLSv1.0,TLSv1.1 and TLSv1.2 from
VMware Knowledge Base article 2148819.
For a list of VMware products supported for TLSv1.0 disablement and the use of TLSv1.1/1.2, consult
VMware Knowledge Base article 2145796.
Virtual SAN Performance: Multiple fixes are introduced in this VMware ESXi 6.0 Update 3 release to optimize
I/O path for improved Virtual SAN performance in All Flash and Hybrid configurations:
Log management and storage improvements were made that enable more logs to be stored per byte of
storage. This should significantly improve performance for write-intensive workloads. Because Virtual SAN is a log based file system,
efficient management of log entries is key to preventing unwarranted build up of logs.
In addition to increasing the packing density of the log entries, for scenarios involving large file being deleted while
data services is turned on, Virtual SAN preemptively de-stages data to the capacity tier which efficiently manages the log growth.
The checksum code path is now more efficient.
Earlier Releases of ESXi 6.0
Features and known issues of ESXi 6.0 are described in the release notes for each release. Release notes for earlier releases of ESXi 6.0, are:
VMware ESXi 6.0 is available in the following languages:
- Simplified Chinese
- Traditional Chinese
Components of VMware vSphere 6.0, including vCenter Server, ESXi, the vSphere Web Client, and the vSphere Client do not
accept non-ASCII input.
ESXi, vCenter Server, and vSphere Web Client Version Compatibility
Product Interoperability Matrix provides details about the compatibility of current and earlier
versions of VMware vSphere components, including ESXi, VMware vCenter Server, the vSphere Web Client,
and optional VMware products. Check the VMware
Product Interoperability Matrix also for information about supported management
and backup agents before you install ESXi or vCenter Server.
The vSphere Web Client is packaged with the vCenter Server. You can install the vSphere Client from the VMware vCenter
autorun menu that is part of the modules ISO file.
Hardware Compatibility for ESXi
To view a list of processors, storage devices, SAN arrays, and I/O devices that are compatible with
vSphere 6.0, use the ESXi 6.0 information in the
Device Compatibility for ESXi
To determine which devices are compatible with ESXi 6.0, use the ESXi 6.0 information in the
Some devices are deprecated and no longer supported on ESXi 6.0. During the upgrade process, the device driver is
installed on the ESXi 6.0 host. The device driver might still function on ESXi 6.0, but the device is not supported on ESXi 6.0. For a
list of devices that are deprecated and no longer supported on ESXi 6.0, see KB 2087970.
Third-Party Switch Compatibility for ESXi
VMware now supports Cisco Nexus 1000V with vSphere 6.0. vSphere requires a minimum NX-OS release of 5.2(1)SV3(1.4).
For more information about Cisco Nexus 1000V, see the
Cisco Release Notes.
As in previous vSphere releases, Ciscso Nexus 1000V AVS mode is not supported.
Guest Operating System Compatibility for ESXi
To determine which guest operating systems are compatible with vSphere 6.0, use the
ESXi 6.0 information in the VMware Compatibility Guide.
Virtual Machine Compatibility for ESXi
Virtual machines that are compatible with ESX 3.x and later (hardware version 4) are supported with
ESXi 6.0. Virtual machines that are compatible with ESX 2.x and later (hardware version 3) are not
supported. To use such virtual machines on ESXi 6.0, upgrade the virtual machine compatibility. See the
vSphere Upgrade documentation.
Installation and Upgrades for This Release
Installation Notes for This Release
vSphere Installation and Setup
documentation for guidance about installing and
configuring ESXi and vCenter Server.
Although the installations are straightforward, several subsequent configuration steps are essential.
Read the following documentation:
vSphere 6.0 Recommended Deployment Models
VMware recommends only two deployment models:
vCenter Server with embedded Platform Services Controller. This model is recommended if one or more standalone vCenter Server
instances are required to be deployed in a data center. Replication between these vCenter Server with embedded Platform Services
Controller models are not recommended.
vCenter Server with external Platform Services Controller. This model is recommended only if multiple vCenter Server instances need
to be linked or want to have reduced footprint of Platform Services Controller in the data center. Replication between these
vCenter Server with external Platform Services Controller models are supported.
Read the vSphere Installation and Setup
documentation for guidance on installing and configuring vCenter Server.
Read the Update sequence for vSphere 6.0 and its compatible VMware products for the proper sequence in which vSphere components should be updated.
Also, read KB 2108548
for guidance on installing and configuring vCenter Server.
vCenter Host OS Information
Read the Knowledge Base article KB 2091273.
Backup and Restore for vCenter Server and the vCenter Server Appliance Deployments that Use an External Platform
Although statements in the vSphere
Installation and Setup documentation restrict you from attempting to backup and restore vCenter Server and vCenter
Server Appliance deployments that use an external Platform Services Controller, you can perform this task by following the
steps in KB 2110294.
Migration from Embedded Platform Services Controller to External Platform Services Controller
vCenter Server with embedded Platform Services Controller cannot be migrated automatically to vCenter Server with external
Platform Services Controller. Testing of this migration utility is not complete.
Before installing vCenter Server, determine your desired deployment option. If more than one vCenter Servers are required for
replication setup, always deploy vCenter with external Platform Services Controller.
Migrating Third-Party Solutions
For information about upgrading with third-party customizations, see the
documentation. For information about using Image Builder to make a custom ISO, see the
vSphere Installation and Setup
Upgrades and Installations Disallowed for Unsupported CPUs
vSphere 6.0 supports only processors available after June (third quarter) 2006. Comparing the processors supported by
vSphere 5.x, vSphere 6.0 no longer supports the following processors:
- AMD Opteron 12xx Series
- AMD Opteron 22xx Series
- AMD Operton 82xx Series
During an installation or upgrade, the installer checks the compatibility of the host CPU with vSphere 6.0.
If your host hardware is not compatible, a purple screen appears with an incompatibility information message,
and the vSphere 6.0 installation process stops.
Upgrade Notes for This Release
For instructions about upgrading vCenter Server and ESX/ESXi hosts, see the
vSphere Upgrade documentation.
Open Source Components for VMware vSphere 6.0
The copyright statements and licenses applicable to the open source software components distributed in
vSphere 6.0 are available at http://www.vmware.com. You need to log in to your
My VMware account. Then, from the Downloads
menu, select vSphere. On the Open Source tab,
you can also download the source files for any GPL, LGPL, or other similar licenses that require the source code or modifications to source code
to be made available for the most recent available release of vSphere.
Product Support Notices
vCenter Server database. Oracle 11g and 12c as an external database for vCenter Server Appliance has been deprecated in
the vSphere 6.0 release. VMware continues to support Oracle 11g and 12c as an external database in vSphere 6.0. VMware will drop support
for Oracle 11g and 12c as an external database for vCenter Server Appliance in a future major release.
vSphere Web Client. The Storage Reports selection from an object's Monitor tab is no longer available
in the vSphere 6.0 Web Client.
vSphere Client. The Storage Views tab is no longer available in the
vSphere 6.0 Client.
Site Recovery Manager: Site Recovery Manager (SRM) versions older than SRM 6.5 do not support IP customization and in-guest callout operations for VMs that are placed on ESXi 6.0 and use VMware Tools version 10.1 and above. For further details, see VMware Tools Issues.
Patches Contained in this Release
This release contains all bulletins for ESXi that were released earlier to the release date of this product. See the VMware Download Patches page for more information about the individual bulletins.
Patch Release ESXi600-Update03 contains the following individual bulletins:
Patch Release ESXi600-Update03 (Security-only build) contains the following individual bulletins:
Patch Release ESXi600-Update03 contains the following image profiles:
Patch Release ESXi600-Update03 (Security-only build) contains the following image profiles:
For information on patch and update classification, see KB 2014447.
The resolved issues are grouped as follows.
CIM and API Issues
High read load of VMware Tools ISO images might cause corruption of flash media
In VDI environment, the high read load of the VMware Tools images can result in corruption of the flash media.
This issue is resolved in this release.
You can copy all the VMware Tools data into its own ramdisk. As a result, the data can be read from the flash media only once per boot. All other reads will go to the ramdisk. vCenter Server Agent (vpxa) accesses this data through the /vmimages directory which has symlinks that point to productLocker.
To activate this feature, follow the steps:
- Use the command to set the advanced ToolsRamdisk option to 1:
esxcli system settings advanced set -o /UserVars/ToolsRamdisk -i 1
- Reboot the host.
Upgrading VMware Tools on multiple VMs might fail
Attempts to upgrade VMware Tools on multiple VMs simultaneously through Update Manager might fail. If this issue occurs, VMware Tools for some VMs might not be upgraded.
This issue is resolved in this release.
Enumeration of SMX provider classes might fail
When you compile HPE ESXi WBEM provider with the 6.0 CIMPDK and install it on an ESXi 6.0 U3 system, enumeration of
SMX provider classes might fail.
This failure might result from enumeration of the following SMX classes, among others:
The following error is displayed by the sfcbd for these SMX classes:
# enum_instances SMX_EthernetPort root/hpq
error: enumInstances Server returned nothing (no headers, no data)
Providers respond to enumeration queries and successfully deliver responses to the sfcbd. There are no provider
restarts or provider core dumps. Each enumeration produces sfcbd CIMXML core dumps, such as
This issue is resolved in this release.
ARP request packets might drop
ARP request packets between two VMs might be dropped if one VM is configured with guest VLAN tagging and the other VM is configured with virtual switch VLAN tagging, and VLAN offload is turned off on the VMs.
Update to the Likewise Kerberos
The Likewise Kerberos is updated to version 1.14.
Update to OpenSSL
The OpenSSL is updated to version openssl-1.0.2j.
Update to PAM
The PAM is updated to version 1.3.0.
Update to the libPNG library
The libPNG library is updated to libpng-1.6.26.
Update to the NTP package
The ESXi NTP package is updated to version 4.2.8p9.
Server Configuration Issues
Update to the libcurl library
The ESXi userworld libcurl library is updated to libcurl- 7.51.0.
ESXi 6.x hosts stop responding after running for 85 days
When this problem occurs, the /var/log/vmkernel log file displays entries similar to the following:
YYYY-MM-DDTHH:MM:SS.833Z cpu58:34255)qlnativefc: vmhba2(5:0.0): Recieved a PUREX IOCB woh oo
YYYY-MM-DDTHH:MM:SS.833Z cpu58:34255)qlnativefc: vmhba2(5:0.0): Recieved the PUREX IOCB.
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674)qlnativefc: vmhba2(5:0.0): sizeof(struct rdp_rsp_payload) = 0x88
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674qlnativefc: vmhba2(5:0.0): transceiver_codes = 0x3
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674)qlnativefc: vmhba2(5:0.0): transceiver_codes[0,1] = 0x3, 0x40
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674)qlnativefc: vmhba2(5:0.0): Stats Mailbox successful.
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674)qlnativefc: vmhba2(5:0.0): Sending the Response to the RDP packet
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674 0 1 2 3 4 5 6 7 8 9 Ah Bh Ch Dh Eh Fh
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 53 01 00 00 00 00 00 00 00 00 04 00 01 00 00 10
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) c0 1d 13 00 00 00 18 00 01 fc ff 00 00 00 00 20
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 00 00 00 00 88 00 00 00 b0 d6 97 3c 01 00 00 00
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 0 1 2 3 4 5 6 7 8 9 Ah Bh Ch Dh Eh Fh
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 02 00 00 00 00 00 00 80 00 00 00 01 00 00 00 04
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 18 00 00 00 00 01 00 00 00 00 00 0c 1e 94 86 08
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 0e 81 13 ec 0e 81 00 51 00 01 00 01 00 00 00 04
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 2c 00 04 00 00 01 00 02 00 00 00 1c 00 00 00 01
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 00 00 00 00 40 00 00 00 00 01 00 03 00 00 00 10
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 50 01 43 80 23 18 a8 89 50 01 43 80 23 18 a8 88
YYYY-MM-DDTHH:MM:SS.833Z cpu58:33674) 00 01 00 03 00 00 00 10 10 00 50 eb 1a da a1 8f
This issue is caused by a qlnativefc driver bug sending a Read Diagnostic Parameters (RDP) response to the HBA adapter with an incorrect transfer length. As a result, the HBA adapter firmware does not free the buffer pool space. Once the buffer pool is exhausted, the HBA adapter is not able to further process any requests causing the HBA adapter to become unavailable. By default, the RDP routine is initiated by the FC Switch and occurs once every hour, resulting in the buffer pool being exhausted in approximately 80 to 85 days under normal circumstances.
This issue is resolved in this release.
Upgrade and Installation Issues
In vSphere 6.0, the HostMultipathStateInfoPath object of the Storage Policy API provides path value as Run Time Name vmhbaX:CX:TX:LX
In ESXi 5.5, HostMultipathStateInfoPath provided path information in this format: HostWWN-ArrayWWN-LUN_ID, For example, sas.500605b0072b6550-sas.500c0ff1b10ea000-naa.600c0ff0001a20bd1887345701000000. However, in ESXi 6.0, the path value appears as vmhbaX:CX:TX:LX, which might impact users who rely on the HostMultipathStateInfoPath object to retrieve information such as HostWWN and ArrayWWN.
This issue is resolved in this release. The HostMultipathStateInfoPath object now displays the path information as Run Time Name and HostWWN-ArrayWWN-LUN_ID.
You can also use the esxcli storage core path list command to retrieve the path related information. This command provides the HostWWN and ArrayWWN details. For more information, see the Knowledge Base article 1003973.
vCenter Server, vSphere Web Client, and vSphere Client Issues
Virtual Machine Management Issues
Virtual SAN Issues
Hostd fails when you upgrade ESXi 5.5.x hosts to ESXi 6.0.x with the ESXi 6.0 patch ESXi600-201611011 or higher
You can observe this issue when you have installed an asynchronous HPSA driver that supports HBA mode. Although ESXi supports getting HPSA disk location information in HBA mode, problems might occur when one of the following conditions is met:
- You installed an old hpssacli utility, version 18.104.22.168 or older.
- You used an external array to connect the HPSA controller.
These problems lead to hostd failures and the host becoming unreachable by vSphere Client and vCenter Server.
This issue is resolved in this release. When you now use the esxcli command to get the disk location information, hostd does not fail. The esxcli command returns an error message similar to the following:
# esxcli storage core device physical get -d naa.500003963c888808 Plugin lsu-hpsa-plugin cannot get information for device with name naa.500003963c888808. Error was: Invalid output for physicaldrive.
VMware HA and Fault Tolerance Configuration Issues
DOM module fails to initialize
Description: The Cluster Level Object Manager Daemon (CLOMD) might not use Virtual SAN on an ESXi host with large number of physical CPUs. This issue can occur if the Virtual SAN DOM module fails to initialize when joining a cluster.
An error message similar to the following is displayed in the clomd.log file:
2016-12-01T22:34:49.446Z 2567759 Failed to run VSI SigCheck: Failure
2016-12-01T22:34:49.446Z 2567759 main: Clomd is starting
2016-12-01T22:34:49.446Z 2567759 main: Is in stretched cluster mode? No
2016-12-01T22:34:49.446Z 2567759 CLOMSetOptions: Setting forground to TRUE
2016-12-01T22:34:49.446Z 2567759 CLOMSetOptions: No default configuration specified.
2016-12-01T22:34:49.447Z 2567759 main: Starting CLOM trace
2016-12-01T22:34:49.475Z 2567759 Cannot open DOM device /dev/dom: No such file or directory
2016-12-01T22:34:49.475Z 2567759 Cannot connect to DOM: Failure
2016-12-01T22:34:49.475Z 2567759 CLOM_CleanupRebalanceContext: Cleaning up rebalancing state
2016-12-01T22:34:49.481Z 2567759 Failed to dump data
2016-12-01T22:34:49.481Z 2567759 main: clomd exit
This issue is resolved in this release.
The known issues existing in ESXi 6.0 are grouped as follows:
New known issues documented in this release are highlighted as New Issue.
DNS suffix might persist even after you change the default configuration in DCUI
An ESXi host might automatically get configured with the default DNS + DNS suffix on first boot, if deployed on a network served by a DHCP server. When you attempt to change the DNS suffix, the DCUI does not remove the existing DNS suffix but just adds the new suffix provided as well.
Workaround: When configuring DNS hostname of the witness OVF, set the FULL FQDN name in the DNS Hostname field to append the correct DNS suffix. You can then remove unwanted DNS suffixes in the Custom DNS Suffix field.
The VMware Tools service user processes might not run on Linux OS after installing the latest VMware Tools package
On Linux OS, you might encounter VMware Tools upgrade/installation issues or the VMware Tools service (vmtoolsd) user processes might not run after installing the latest VMware Tools package. The issue occurs if the your glibc version is older than version 2.5, like SLES10sp4.
Workaround: Upgrade the Linux glibc to version 2.5 or above.
Review also the Installation Issues section of the release notes. Many installation issues can also impact your upgrade process.
vCenter Single Sign-On and Certificate Management Issues
Attempts to upgrade from ESXi 6.x to 6.0 Update 2 and above with the esxcli software vib update command fail
Attempts to upgrade from ESXi 6.x to 6.0 Update 2 with the esxcli software vib update fails with error messages similar to the following:
VIB VMware_bootbank_esx-base_6.0.0-2.34.xxxxxxx requires vsan << 6.0.0-2.35, but the requirement cannot be satisfied within the ImageProfile.
VIB VMware_bootbank_esx-base_6.0.0-2.34.xxxxxxx requires vsan >= 6.0.0-2.34, but the requirement cannot be satisfied within the ImageProfile.
The issue occurs due to introduction of a new Virtual SAN VIB which is interdependent with the esx-base VIB and the esxcli software vib update command only updates the VIBs already installed on the system.
Workaround: To resolve this issue, run the esxcli software profile update as shown in the following example:
esxcli software profile update -d /vmfs/volumes/datastore1/update-from-esxi6.0-6.0_update02.zip -p ESXi-6.0.0-20160302001-standard
SSLv3 remains enabled on Auto Deploy after upgrade from earlier release of vSphere 6.0 to vSphere 6.0 Update 1 and above
When you upgrade from an earlier release of vSphere 6.0 to vSphere 6.0 Update 1 and above, the SSLv3 protocol remains enabled on Auto Deploy.
Workaround: Perform to the following steps to disable SSLv3 using PowerCLI commands:
- Run the following command to Connect to vCenter Server:
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> Connect-VIServer -Server <FQDN_hostname or IP Address of vCenter Server>
- Run the following command to check the current sslv3 status:
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> Get-DeployOption
- Run the following command to disable sslv3:
PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> Set-DeployOption disable-sslv3 1
- Restart the Auto Deploy service to update the change.
- Fibre Channel host bus adapter device number might change after ESXi upgrade from 5.5.x to 6.0
During ESXi upgrade from 5.5.x to 6.0, the Fibre Channel host bus adapter device number changes occasionally. The device number might change to another number if you use the
esxcli storage core adapter list command.
For example, the device numbers for a Fibre Channel host bus adapter might look similar to the following before ESXi upgrade:
The device numbers from the Fibre Channel host bus adapter might look similar to the following after an ESXi upgrade 6.0:
The example illustrates the random change that might occur if you use the
esxcli storage core adapter list command: the device alias numbers vmhba2 and vmhba3 change to vmhba64 and vmhba65, while device numbers vmhba5 and vmhba6 are not changed. However, if you used the
esxcli hardware pci list command, the device numbers do not change after upgrade.
This problem is external to VMware and may not affect you. ESXi displays device alias names but it does not use them for any operations. You can use the host profile to reset the device alias name. Consult VMware product documentation and knowledge base articles.
Active Directory settings are not retained post-upgrade
The Active Directory settings configured in the ESXi host before upgrade are not retained when the host is upgraded to ESXi 6.0.
Workaround: Add the host to the Active Directory Domain after upgrade if the pre-upgrade ESXi version is 5.1 or later. Do not add the host to the Active Directory Domain after upgrade if the pre-upgrade ESXi version is ESXi 5.0.x.
After ESXi upgrade to 6.0 hosts that were previously added to the domain are no longer joined to the domain
When upgrading to from vSphere 5.5 to vSphere 6.0 for the first time, the Active Directory configuration is not retained.
Workaround: After upgrade, rejoin the hosts to the vCenter Server domain:
Add the hosts to vCenter Server.
Join the hosts to domain (for example, example.com)
Upgrade all the hosts to ESXi 6.0.
Manually join one recently upgraded host to domain.
Extract the host profile and disabled all other profiles except Authentication.
Apply the manually joined host profile to the other recently upgraded hosts.
Previously running VMware ESXi Dump Collector service resets to default Disabled setting after upgrade of vCenter Server for Windows
The upgrade process installs VMware Vsphere ESXi Dump Collector 6.0 as part of a group of optional services for vCenter Server. You must manually enable the VMware vSphere ESXi Dump Collector service to use it as part of vCenter Server 6.0 for Windows.
Workaround: Read the VMware documentation or search the VMware Knowledge Base for information on how to enable and run optional services in vCenter Server 6.0 for Windows.
Enable the VMware vSphere ESXi Dump Collector service in the operating system:
In the Control Panel menu, select Administrative Tools and double-click on Services.
Right click VMware vSphere ESXi Dump Collector and Edit Startup Type.
Set the Start-up Type to Automatic.
Right Click VMware vSphere ESXi Dump Collector and Start.
The Service Start-up Type is set to automatic and the service is in a running state.
Cannot connect to VM console after SSL certificate upgrade of ESXi host
A certificate validation error might result if you upgrade the SSL certificate that is used by an ESXi host, and you then attempt to connect to the VM console of any VM running when the certificate was replaced. This is because the old certificate is cached, and any new console connection is rejected due to the mismatch.
The console connection might still succeed, for example, if the old certificate can be validated through other means, but is not guaranteed to succeed. Existing virtual machine console connections are not affected, but you might see the problem if the console was running during the certificate replacement, was stopped, and was restarted.
Workaround: Place the host in maintenance mode or suspend or power off all VMs. Only running VMs are affected.
As a best practice, perform all SSL certificate upgrades after placing the host in maintenance mode.
Certain vSphere functionality does not support IPv6
You can enable IPv6 for all nodes and components except for the following features:
IPv6 addresses for ESXi hosts and vCenter Server that are not mapped to fully qualified domain names (FQDNs) on the DNS server.
Use FQDNs or make sure the IPv6 addresses are mapped to FQDNs on the DNS servers for reverse name lookup.
PXE booting as a part of Auto Deploy and Host Profiles
Workaround: PXE boot an ESXi host over IPv4 and configure the host for IPv6 by using Host Profiles.
Connection of ESXi hosts and the vCenter Server Appliance to Active Directory
Workaround: Use Active Directory over LDAP as an identity source in vCenter Single Sign-On.
NFS 4.1 storage with Kerberos
Workaround: Use NFS 4.1 with AUTH_SYS.
Connection of the vSphere Management Assistant and vSphere Command-Line Interface to Active Directory.
Workaround: Connect to Active Directory over LDAP.
Use of the vSphere Client to enable IPv6 on vSphere features
Workaround: Use the vSphere Web Client to enable IPv6 for vSphere features.
Recursive panic might occur when using ESXi Dump Collector
Recursive kernel panic might occur when the host is in panic state while it displays the purple diagnostic screen and write the core dump over the network to the ESXi Dump Collector. A VMkernel zdump file might not be available for troubleshooting on the ESXi Dump Collector in vCenter Server.
In the case of a recursive kernel panic, the purple diagnostic screen on the host displays the following message:
2014-09-06T01:59:13.972Z cpu6:38776)Starting network coredump from host_ip_address to esxi_dump_collector_ip_address.
[7m2014-09-06T01:59:13.980Z cpu6:38776)WARNING: Net: 1677: Check what type of stack we are running on [0m
Recursive panic on same CPU (cpu 6, world 38776, depth 1): ip=0x418000876a27 randomOff=0x800000:
#GP Exception 13 in world 38776:vsish @ 0x418000f0eeec
Secondary panic trap frame registers:
RAX:0x0002000001230121 RCX:0x000043917bc1af80 RDX:0x00004180009d5fb8 RBX:0x000043917bc1aef0
RSP:0x000043917bc1aee8 RBP:0x000043917bc1af70 RSI:0x0002000001230119 RDI:0x0002000001230121
R8: 0x0000000000000038 R9: 0x0000000000000040 R10:0x0000000000010000 R11:0x0000000000000000
R12:0x00004304f36b0260 R13:0x00004304f36add28 R14:0x000043917bc1af20 R15:0x000043917bc1afd0
CS: 0x4010 SS: 0x0000 FS: 0x4018 GS: 0x4018 IP: 0x0000418000f0eeec RFG:0x0000000000010006
2014-09-06T01:59:14.047Z cpu6:38776)Backtrace for current CPU #6, worldID=38776, rbp=0x43917bc1af70
2014-09-06T01:59:14.056Z cpu6:38776)0x43917bc1aee8:[0x418000f0eeec]firstname.lastname@example.orgAPI#9.2+0x4 stack: 0x0, 0x43a18b4a5880,
2014-09-06T01:59:14.068Z cpu6:38776)Recursive panic on same CPU (cpu 6, world 38776): ip=0x418000876a27 randomOff=0x800000:
#GP Exception 13 in world 38776:vsish @ 0x418000f0eeec
Halt$Si0n5g# PbC8PU 7.
Recursive kernel panic might occur when the VMkernel panics while heavy traffic is passing through the physical network adapter that is also configured to send the core dumps to the collector on vCenter Server.
Workaround: Perform either of the following workarounds:
Dedicate a physical network adapter to core dump transmission only to reduce the impact from system and virtual machine traffic.
Disable the ESXi Dump Collector on the host by running the following ESXCLI console command:
esxcli system coredump network set --enable false
NFS Version 4.1 Issues
Virtual machines on an NFS 4.1 datastore fail after the NFS 4.1 share recovers from an all paths down (APD) state
When the NFS 4.1 storage enters an APD state and then exits it after a grace period, powered on virtual machines that run on the NFS 4.1 datastore fail. The grace period depends on the array vendor.
After the NFS 4.1 share recovers from APD, you see the following message on the virtual machine summary page in the vSphere Web Client:
The lock protecting VM.vmdk has been lost, possibly due to underlying storage issues. If this virtual machine is configured to be highly available, ensure that the virtual machine is running on some other host before clicking OK.
After you click OK, crash files are generated and the virtual machine powers off.
NFS 4.1 client loses synchronization with server when trying to create new sessions
After a period of interrupted connectivity with the server, the NFS 4.1 client might lose synchronization with the server when trying to create new sessions. When this occurs, the
vmkernel.log file contains a throttled series of warning messages noting that an NFS41 CREATE_SESSION request failed with NFS4ERR_SEQ_MISORDERED.
Workaround: Perform the following sequence of steps.
Attempt to unmount the affected file systems. If no files are open when you unmount, this operation succeeds and the NFS client module cleans up its internal state. You can then remount the file systems that were unmounted and resume normal operation.
Take down the NICs connecting to the mounts' IP addresses and leave them down long enough for several server lease times to expire. Five minutes should be sufficient. You can then bring the NICs back up. Normal operation should resume.
If the preceding steps fail, reboot the ESXi host.
NFS 4.1 client loses synchronization with an NFS server and connection cannot be recovered even when session is reset
After a period of interrupted connectivity with the server, the NFS 4.1 client might lose synchronization with the server and the synchronized connection with the server cannot be recovered even if the session is reset. This problem is caused by an EMC VNX server issue. When this occurs, the
vmkernel.log file contains a throttled series of warning messages noting that NFS41: NFS41ProcessSessionUp:2111: resetting session with mismatched clientID; probable server bug.
Workaround: To end the session, unmount all datastores and then remount them.
ONTAP Kerberos volumes become inaccessible or experience VM I/O failures
A NetApp server does not respond when it receives RPCSEC_GSS requests that arrive out of sequence. As a result, the corresponding I/O operation stalls unless it is terminated and the guest OS can stall or encounter I/O errors. Additionally, according to RFC 2203, the client can only have a number of outstanding requests equal to seq_window (32 in case of ONTAP) according to RPCSEC_GSS context and it must wait until the lowest of these outstanding requests is completed by the server. Therefore, the server never replies to the out-of-sequence RPCSEC_GSS request, and the client stops sending requests to the server after it reaches the maximum seq_window number of outstanding requests. This causes the volume to become inaccessible.
Workaround: None. Check the latest Hardware Compatibility List (HCL) to find a supported ONTAP server that has resolved this problem.
You cannot create a larger than 1 TB virtual disk on NFS 4.1 datastore from EMC VNX
NFS version 4.1 storage from EMC VNX with firmware version 7.x supports only 32-bit file formats. This prevents you from creating virtual machine files that are larger than 1 TB on the NFS 4.1 datastore.
Workaround: Update the EMC VNX array to version 8.x.
NFS 4.1 datastores backed by EMC VNX storage become inaccessible during firmware upgrades
When you upgrade EMC VNX storage to a new firmware, NFS 4.1 datastores mounted on the ESXi host become inaccessible. This occurs because the VNX server changes its major device number after the firmware upgrade. The NFS 4.1 client on the host does not expect the major number to change after it has established connectivity with the server, and causes the datastores to be permanently inaccessible.
Workaround: Unmount all NFS 4.1 datastores exported by the VNX server before upgrading the firmware.
When ESXi hosts use different security mechanisms to mount the same NFS 4.1 datastore, virtual machine failures might occur
If different ESXi hosts mount the same NFS 4.1 datastore using different security mechanisms, AUTH_SYS and Kerberos, virtual machines placed on this datastore might experience problems and failure. For example, your attempts to migrate the virtual machines from host1 to host2 might fail with permission denied errors. You might also observe these errors when you attempt to access a host1 virtual machine from host2.
Workaround: Make sure that all hosts that mount an NFS 4.1 volume use the same security type.
Attempts to copy read-only files to NFS 4.1 datastore with Kerberos fail
The failure might occur when you attempt to copy data from a source file to a target file. The target file remains empty.
When you create a datastore cluster, uniformity of NFS 4.1 security types is not guaranteed
While creating a datastore cluster, vSphere does not verify and enforce the uniformity of NFS 4.1 security types. As a result, datastores that use different security types, AUTH_SYS and Kerberos, might be a part of the same cluster. If you migrate a virtual machine from a datastore with Kerberos to a datastore with AUTH_SYS, the security level for the virtual machine becomes lower.
This issue applies to such functionalities as vMotion, Storage vMotion, DRS, and Storage DRS.
Workaround: If Kerberos security is required for your virtual machines, make sure that all NFS 4.1 volumes that compose the same cluster use only the Kerberos security type. Do not include NFS 3 datastores, because NFS 3 supports only AUTH_SYS.
Virtual SAN Issues
New Issue Virtual SAN Health UI does not show because of a timeout
When you access the Virtual SAN Health UI under a Virtual SAN cluster > Monitor > Virtual SAN > Health, the UI does not show. A possible cause is the vSphere ESX Agent Manager hanging and resulting in a timeout. To confirm, open the Virtual SAN health log located at /var/log/vmware/vsan-health/vmware-vsan-health-service.log and search for calls to the vSphere ESX Agent Manager service by using the string VsanEamUtil.getClusterStatus:.
Workaround: Restart the vSphere ESX Agent Manager service by using the vSphere Web Client and refresh the Virtual SAN health UI.
New Issue Virtual SAN disk serviceability does not work when you use third-party lsi_msgpt3 drivers
A health check of a two or three-node Virtual SAN cluster under Virtual SAN cluster > Monitor > Virtual SAN > Health > Limit Health > after 1 additional host failure shows red and triggers a false vCenter Server event or alarm when the disk space usage of the cluster exceeds 50%.
Workaround: Add one or more hosts to the Virtual SAN cluster or add more disks to decrease the disk space usage of the cluster under 50%.
New Issue The limit health check of a two or three-node Virtual SAN cluster shows red
The plugin for Virtual SAN disk serviceability, lsu-lsi-lsi-msgpt3-plugin, supports the operation to get the device location and turn on or off the disk LED. The VMware lsi_msgpt3 inbox driver supports the serviceability plugin. However, if you use a third-party asynchronous driver, the plugin does not work.
Workaround: Use the VMware inbox lsi_msgpt3 driver version 06.255.10.00-2vmw or later.
Virtual Volumes Issues
Failure to create virtual datastores due to incorrect certificate used by Virtual Volumes VASA provider
Occasionally, a self-signed certificate used by the Virtual Volumes VASA provider might incorrectly define the
KeyUsage extension as critical without setting the
keyCertSign bit. In this case, the provider registration succeeds. However, you are not able to create a virtual datastore from storage containers reported by the VASA provider.
Workaround: Self-signed certificate used by the VASA provider at the time of provider registration should not define
KeyUsage extension as critical without setting the
General Storage Issues
Server Configuration Issues
ESXi 6.0 Update 2 hosts connected to certain storage arrays with a particular version of the firmware might see I/O timeouts and subsequent aborts
When ESXi 6.0 Update 2 hosts connected to certain storage arrays with a particular version of the firmware send requests for SMART data to the storage array, and if the array responds with a PDL error, the PDL response behavior in 6.0 update 2 might result in a condition where these failed commands are continuously retried thereby blocking other commands. This error results in widespread I/O timeouts and subsequent aborts.
Also, the ESXi hosts might take a long time to reconnect to the vCenter Server after reboot or the hosts might go into a Not Responding state in the vCenter Server. Storage-related tasks such as HBA rescan might take a very long time to complete.
Workaround: To resolve this issue, see Knowledge Base article 2133286.
vSphere Web Client incorrectly displays Storage Policy as attached when new VM is created from an existing disk
When you use the vSphere Web Client to create a new VM from an existing disk and specify a storage policy when setting up the disk. The filter appears to be attached when you select the new VM --> click on VM policies --> Edit VM storage policies, however the filter is not actually attached. You can check the .vmdk file or the vmkfstools --iofilterslist <vmdk-file>' to verify if the filter is attached or not.
Workaround: After you create the new VM, but before you power it on, add the filter to the vmdk by clicking on VM policies --> Edit VM storage policies.
NFS Lookup operation returns NFS STALE errors
When you deploy large number of VMs in the NFS datastore, the VM deployment fails with an error message similar to the following due to a race condition:
Stale NFS file handle
Workaround: Restart the Lookup operation. See Knowledge Based article 2130593 for details.
Attempts to create a VMFS datastore on Dell EqualLogic LUNs fail when QLogic iSCSI adapters are used
You cannot create a VMFS datastore on a Dell EqualLogic storage device that is discovered through QLogic iSCSI adapters.
When your attempts fail, the following error message appears on vCenter Server:
Unable to create Filesystem, please see VMkernel log for more details: Connection timed out.
The VMkernel log contains continuous
iscsi session blocked and
iscsi session unblocked messages.
On the Dell EqualLogic storage array, monitoring logs show a
protocol error in packet received from the initiator message for the QLogic initiator IQN names.
This issue is observed when you use the following components:
Dell EqualLogic array firmware : V6.0.7
QLogic iSCSI adapter firmware versions : 3.00.01.75
Driver version : 5.01.03.2-7vmw-debug
Workaround: Enable the iSCSI ImmediateData adapter parameter on QLogic iSCSI adapter. By default, the parameter is turned off.
You cannot change this parameter from the vSphere Web Client or by using esxcli commands. To change this parameter, use the vendor provided software, such as QConvergeConsole CLI.
ESXi host with Emulex OneConnect HBA fails to boot
When an ESXi host has the Emulex OneConnect HBA installed, the host might fail to boot. This failure occurs due to a problem with the Emulex firmware.
Workaround: To correct this problem, contact Emulex to get the latest firmware for your HBA.
If you continue to use the old firmware, follow these steps to avoid the boot failure:
When ESXi is loading, press Shift+O before booting the ESXi kernel.
Leave the existing boot option as is, and add a space followed by
Flash Read Cache does not accelerate I/Os during APD
When the flash disk configured as a virtual flash resource for Flash Read Cache is faulty or inaccessible, or the disk storage is unreachable from the host, the Flash Read Cache instances on that host are invalid and do not work to accelerate I/Os. As a result, the caches do not serve stale data after connectivity is re-established between the host and storage. The connectivity outage might be temporary, all paths down (APD) condition, or permanent, permanent device loss (PDL). This condition persists until the virtual machine is power-cycled.
Workaround: The virtual machine can be power-cycled to restore I/O acceleration using Flash Read Cache.
All Paths Down (APD) or path-failovers might cause system failure
In a shared SAS environment, APD or path-failover situations might cause system failure if the disks are claimed by the lsi_msgpt3 driver and they are experiencing heavy I/O activity.
Frequent use of SCSI abort commands can cause system failure
With heavy I/O activity, frequent SCSI abort commands can cause a very slow response from the MegaRAID controller. If an unexpected interrupt occurs with resource references that were already released in a previous context, system failure might result.
iSCSI connections fail and datastores become inaccessible when IQN changes
This problem might occur if you change the IQN of an iSCSI adapter while iSCSI sessions on the adapter are still active.
Workaround: When you change the IQN of an iSCSI adapter, no session should be active on that adapter. Remove all iSCSI sessions and all targets on the adapter before changing the IQN.
nvmecli online and offline operations might not always take effect
When you perform the
nvmecli device online -A vmhba* operation to bring a NVMe device online, the operation appears to be successful. However, the device might still remain in offline state.
Workaround: Check the status of NVMe devices by running the
nvmecli device list command.
VMware HA and Fault Tolerance Issues
Remediation fails when applying a host profile from a stateful host to a host provisioned with Auto Deploy
When applying a host profile from a statefully deployed host to a host provisioned with Auto Deploy (stateless host) with no local
storage, the remediation attempt fails with one of the following error messages:
The vmhba device at PCI bus address sxxxxxxxx.xx is not present on your host. You must shut down and then insert a card into PCI slot yy.
The type of card should exactly match the one in the reference host.
No valid coredump partition found.
Workaround: Disable the plug-in that is causing the issue (for example, the Device Alias Configuration or Core Dump Configuration)
from the host profile, and then remediate the host profile.
Applying host profile with static IP to a host results in compliance error
If you extract a host profile from a host with a DHCP network configuration, and then edit the host profile to have
a static IP address, a compliance error occurs with the following message when you
apply it to another host:
Number of IPv4 routes did not match.
Workaround: Before extracting the host profile from the DHCP host, configure the host so that it has a static IP
- When you hot-add a virtual network adapter that has network resources overcommitted, the virtual machine might be powered off
On a vSphere Distributed Switch that has Network I/O Control enabled, a powered on virtual machine is configured with a bandwidth reservation according to the reservation for virtual machine system traffic on the physical network adapter on the host. You hot-add a network adapter to the virtual machine setting network bandwidth reservation that is over the bandwidth available on the physical network adapters on the host.
When you hot-add the network adapter, the VMkernel starts a Fast Suspend and Resume (FSR) process. Because the virtual machine requests more network resources than available, the VMkernel exercises the failure path of the FSR process. A fault in this failure path causes the virtual machine to power off.
Workaround: Do not configure bandwidth reservation when you add a network adapter to a powered on virtual machine.
Guest Operating System Issues
Legacy Fault Tolerance (FT) not supported on Intel Skylake-DT/S, Broadwell-EP, Broadwell-DT, and Broadwell-DE platform
Legacy FT is not supported on Intel Skylake-DT/S, Broadwell-EP, Broadwell-DT, and Broadwell-DE platform. Attempts to power on a virtual machine will fail after you enable single-processor, Legacy Fault Tolerance.
Supported Hardware Issues
Attempts to enable passthrough mode on NVMe PCIe SSD devices might fail after hot plug
To enable passthrough mode on an SSD device from the vSphere Web Client, you select a host, click
the Manage tab, click Settings, navigate to the Hardware section,
click PCI Devices > Edit, select a device from a list of active devices that can be
enabled for passthrough, and click OK. However, when you hot plug a new NVMe device to an ESXi 6.0 host that does not have a
PCIe NVMe drive, the new NVMe PCIe SSD device cannot be enabled for passthrough mode and does not appear in the list of
available passthrough devices.
Workaround: Restart your host. You can also run the command on your ESXi host.
Log in as a root user.
Run the command
CIM and API Issues
When you run esxcli to get the disk location, the result is not correct for Avago controllers on HP servers
When you run
esxcli storage core device physical get, against an Avago controller on an HP server, the result is not correct.
For example, if you run:
esxcli storage core device physical get -d naa.5000c5004d1a0e76
The system returns:
Physical Location: enclosure 0, slot 0
The actual label of that slot on the physical server is 1.
Workaround: Check the slot on your HP server carefully. Because the slot numbers on the HP server start at 1, you have to increase the slot number that the command returns for the correct result.
The sfcb-vmware_raw might fail
The sfcb-vmware_raw might fail as the maximum default plugin resource group memory allocated is not enough.
Workaround: Add UserVars CIMOemPluginsRPMemMax for memory limits of sfcbd plugins using the following command and restart the sfcbd for the new plugins value to take effect:
esxcfg-advcfg -A CIMOemPluginsRPMemMax --add-desc 'Maximum Memory for plugins RP' --add-default XXX --add-type int --add-min 175 --add-max 500
XXX being the memory limit you want to allocate. This value should be within the minimum (175) and maximum (500) values.