VMware Integrated OpenStack 2.5.1 Release Notes

VMware Integrated OpenStack 2.5.1 | 4 OCTOBER 2016 | BUILD 4443700

What's in the Release Notes

The release notes cover the following topics:

About VMware Integrated OpenStack

VMware Integrated OpenStack greatly simplifies deploying an OpenStack cloud infrastructure by streamlining the integration process. VMware Integrated OpenStack delivers out-of-the-box OpenStack functionality and an easy configuration workflow through a deployment manager vApp that runs directly in vCenter.

Internationalization

VMware Integrated OpenStack version 2.5.1 is available in English and seven additional languages: Simplified Chinese, Traditional Chinese, Japanese, Korean, French, German, and Spanish. ASCII characters must be used for all input and naming conventions of OpenStack resources (such as project names, usernames, image names, and so on) and for the underlying infrastructure components (such as ESXi hostnames, vSwitch port group names, data center names, datastore names, vSwitch port group names, and so on).

Compatibility

The VMware Product Interoperability Matrix provides details about the compatibility of the current version of VMware Integrated OpenStack with VMware vSphere components, including ESXi, VMware vCenter Server, the vSphere Web Client, and optional VMware products. Check the VMware Product Interoperability Matrix also for information about supported management and backup agents before you install VMware Integrated OpenStack or other VMware products.

Upgrading to VMware Integrated OpenStack 2.5.1

You perform the upgrade procedure directly in the VMware Integrated OpenStack manager. The complete multi-step procedure is described in detail in the VMware Integrated OpenStack Administration Guide.

Open Source Components for VMware Integrated OpenStack 2.5.1

The copyright statements and licenses applicable to the open source software components distributed in VMware Integrated OpenStack 2.5.1 are available on the Open Source tab of the product download page. You can also download the source files for any GPL, LGPL, or other similar licenses that require the source code or modifications to source code to be made available for the most recent available release of VMware Integrated OpenStack.

Known Issues

VMware Integrated OpenStack 2.5.1 has the following known issues. If you encounter an issue that is not in this known issues list, search the VMware Knowledge Base, or let us know by contacting VMware Technical Support.

  • Unable to modify syslog setting post deployment in VIO Manager interface.
    After deploying VIO, you cannot modify the syslog server configuration using the setting in the VIO Manager interface (VMware Integrated OpenStack > Management Server > Edit Settings > vApp Options).

    Workaround: Modify the configuration here: VMware Integrated OpenStack > OpenStack Cluster > Manage > LogInsight.

  • Dashboard might show a Volume as attached even if it failed to attach.
    This is a known OpenStack issue, first reported in the Icehouse release.

  • Long image upload times cause NotAuthenticated failure.
    This is a known OpenStack issue (https://bugs.launchpad.net/glance/+bug/1371121), first reported in the Icehouse release.

  • OpenStack dashboard does not immediately reflect volume extension.
    If you extend the size of a volume, the dashboard might still display the pre-extended size. Eventually, the correct size appears.

  • Special characters in datastore names not supported by Glance (Image Service).
    If a datastore name has non-alphanumeric characters like colons, ampersands, or commas, the datastore cannot be added to the Glance service. Specifically, the following characters are not permitted in Glance datastore names because their use is reserved for other purposes and therefore can interfere with the configuration: : , / $ (colon, comma, forward slash, dollar). This issue has been fast-tracked for resolution.

  • If either controller VM reboots, high availability might be compromised.
    When a controller fails, the other controller continues to provide services. However, when the initial controller reboots, it might no longer provides services, and thus is not available if the other controller also fails. This issue has been fast-tracked for resolution.

    Workaround: If a controller fails and HA is invoked, review your deployment to ensure that both controllers are providing services after the failed controller reboots.

  • Metadata service is not accessible on subnets created with the no-gateway option.
    In 2.x, autoconfiguration is turned off for Edge VMs. When applicable, DHCP sets the gateway and metadata is served through this gateway Edge. As a result, when a subnet is created with the no-gateway option, there is no router Edge to capture the metadata traffic.

    Workaround: For networks with the no-gateway option, configure a route for 169.254.169.254/32 to forward traffic to DHCP Edge IP.

  • Horizon dashboard shows error after switching projects as admin user
    If you are logged into Horizon as the administrative user and try to switch between projects using the drop-down menu in Horizon, the dashboard might begin returning errors. This is caused by an issue in OpenStack.

    Workaround: Log out of the Horizon dashboard and log back in to restore the dashboard.

  • WaitConditionHandle not working with current heat.conf.
    Using OS::Heat::WaitConditionHandle in a Heat Orchestration Template (HOT) template might generate the following error:

    Resource Create Failed: Error: Resources.Mysql.Resources.Wait Handle: Cannot Get Stack Domain User Token, No Stack Domain Id Configured, Please Fix Your Heat.Conf

    This issue has been fast-tracked for resolution.

  • Upgrade to 2.0 Fails with "No qualified ESXi Host for Controller-1" error
    This error is caused because user customizations to the omjs.properties file are not passed to 2.x during the upgrade process.

    Workaround: Reapply your customizations to the 2.0 omjs.properties file and restart the VMware Integrated OpenStack manager.

  • Reverting from 2.x to 1.x results in error alert.
    If you revert to 1.0.x, VMware Integrated OpenStack might show an authentication error alert.

    Workaround: Close the alert and VMware Integrated OpenStack manager, and restart the vSphere Web client service. When you restart the VMware Integrated OpenStack manager, the VMware Integrated OpenStack service will restart without the error.

  • Problem uploading patch file in Firefox Web Browser.
    If you are using Firefox to update the patch for VMware Integrated OpenStack, the upload will fail if Firefox is using version 19 of the Adobe Flash plugin.
    Per the Adobe Bug Base (https://bugbase.adobe.com/index.cfm?event=bug&id=4037494), this issue affects only the Firefox browser and later versions of Chrome.

    Workaround: Obtain the patch using the CLI. You can also work around this issue by using an alternative browser or restoring the Flash plugin in your Firefox browser to an earlier version (15,16,17 of 18.)

  • OpenStack management service does not automatically restart.
    Under certain conditions, the OpenStack management service does not automatically restart. For example, after a failover event, all OpenStack services successfully restart but the management service remains unreachable.

    Workaround: Manually restart the VMware Integrated OpenStack vApp in the vSphere Web Client. Right-click the icon in the Inventory page and select Shut Down. After all the services shut down, power on the vApp. Check the OpenStack manager logs to confirm that the restart was successful.

    NOTE: Restarting interrupts services. This issue is fast-tracked for resolution in a future VMware Integrated OpenStack release.

  • Empty tenant_id used in creating LBaaS v2.0 resource is not rejected.
    When creating a new resource a new LBaaS resource, such as the health monitor, NSX LBaaS v2.0 accepts empty tenant_id values but OpenStack Neutron LBaaS rejects them. This is a conflict between the Kilo and Mitaka OpenStack releases, and should be resolved when VMware Integrated OpenStack migrates to Mitaka in a future release.

    Workaround: No user action required. As a temporary solution, VMware Integrated OpenStack accepts the new LBaaS resource and applies the tenant_id value of the tenant that creates the resource.

    NOTE: NSX does not support multiple tenants attached to the same subnet. See the NSX product documentation.

  • Recovery operation returns "Nodes already exist" error.
    Under certain conditions, running the viocli recovery - <DB name> command fails if the ansible operation is interrupted. As a result, the database nodes remain and causes the error.

    Workaround: Manually remove the nodes and run the viocli recovery command again.

  • LBaaS v2 migration: health monitors not associated to a pool do not migrate.
    In LBaaS v2, health-monitors are required to specify and be attached to a pool. In LBaaS v1, health monitors can be created without pool association, and associated with a pool in a separate procedure.

    As a result, when migrating to LBaaS v2, unassociated health monitors are excluded.

    Workaround: Before migrating to LBaaS v1, associate all health monitors with a pool to ensure their successful migration. The migration process is optional after installing or upgrading to VMware Integrated OpenStack 2.5. See the VMware Integrated OpenStack Administration Guide.

  • NSX LBaaS v2.0 tenant limitation.
    NSX load balancers support only one tenant per subnet. Under normal operation, this is not an issue because tenants create their own load balancers. If a user attempts to create and attach a load balancer to a subnet, the load balancer will be created in an ERROR state.

    Workaround: Allow tenants to create their own load balancers. Do not create and attach a load balancer to an existing subnet.

  • VMware Integrated OpenStack 2.5 requires Kilo version of Python Heat client.
    Because VMware Integrated OpenStack 2.5 is based on the Kilo release of OpenStack, the Orchestration component requires the Kilo version of the python-heatclient. See https://launchpad.net/python-heatclient/kilo for the required version.

  • Renaming router might result in dropped packets.
    Observed in deployments with NSX 6.2.2. If you rename a shared router while pinging the VM from an external source, some in-traffic packets might be lost due to the backend update request.

    NOTE: This issue is resolved for the VMware Integrated OpenStack 3.0 release.

  • Insufficient disk space for upgrade process.
    The upgrade procedure for VMware Integrated OpenStack 2.5 omits that you must have at least 2 GB of free disk space available, after uploading the upgrade package, to complete the upgrade process.

    Workaround: Before starting the upgrade procedure, you can either extend the disk or clear the /opt/vmware/vio/patches/downloads directory of all .deb files, excluding the 2.5 package.

  • Image import results in "qemu-img convert" error.
    When importing an Ubuntu or other source image into Glance, you must specify the disk format that matches the source image attributes.

    Workaround: Repeat the image import procedure, specifying the correct disk format. See the product documentation for guidance.

    NOTE: In a future VMware Integrated OpenStack release, the interface will be modified to better guide the user to avoid this error during the image import process.

  • Heat stack deletion fails with "Failed to publish configuration on NSX Edge" error.
    Observed in deployments using NSX v6.2.2. Under stressful conditions, the Heat stack or OpenStack API might fail at the backend.

    Workaround: Retry the failed operation.

    NOTE: This issue is fast-tracked for resolution in a future NSX release.

  • Adding subnet returns error message.
    When adding a subnet interface to a private network, you receive this error: "Request Failed: internal server error while processing your request". This indicates that the subnet interface points to a router whose gateway network is an external network that does not have a subnet.

    Workaround: You can resolve this issue by adding a subnet to the external network configuration, then recreating the router that uses that network as a gateway. Reconfigure the private network with the subnet interface to the updated router.

    NOTE: This issue is fast-tracked for resolution in a future VMware Integrated OpenStack release.

  • "nova list" command fails when password is manually entered.
    This is an OpenStack bug documented in detail here: https://bugs.launchpad.net/python-novaclient/+bug/1525378. This issue occurs when the OS_PASSWORD environmental variable for the Nova client is not specified.

    Workaround: Set the OS_PASSWORD environmental variable for the Nova client.

  • Template import fails with "400 Bad Request: 10.0.0.1 must match with vcenter-host.example.com"
    Adding a VM template as a Glance image might fail if the location URI used to create the image refers to the vCenter server using IP address because VMware Integrated OpenStack requires the hostname.

    Workaround: When creating the image, use the hostname for the location URI value.

  • Unable to select flavor with 0 root disk size.

    If you create a flavor where root_disk=0, that flavor might appears as disabled when a user tries to select it when creating an instance in the VMware Integrated OpenStack dashboard (Horizon).

    Workaround: Create the instance using the CLI, which allows you to specify the flavor.

    NOTE: This issue is fast-tracked for resolution in a future VMware Integrated OpenStack release.

  • AD/LDAP configuration validates but keystone fails to start.
    During the installation process, the Active Directory/LDAP configuration validates, but the Keystone service fails to start during actual deployment. This issue is caused because the common name of the certificate does not match with the hostname of the LDAP server.

    Workaround: Prior to installing and deploying VMware Integrated OpenStack, verify that the certificate name and LDAP server hostname match.

    NOTE: This issue is fast-tracked for resolution in a future VMware Integrated OpenStack release.

  • Adding router interface fails with flush error.

    Adding router interface operation on a distributed router might fail due to a race condition with the error message "FlushError: New instance <xx>, with identity key <yy> conflicts with persistent instance <zz>".

    Workaround: Retry the failed operation.

    NOTE This issue is fast-tracked for resolution in the next VMware Integrated OpenStack release.

  • Block storage procedures omit package installation.
    The topic "Configure the Backup Service for Block Storage" in the VMware Integrated OpenStack Administration Guide omits the step of obtaining and installing two debian packages that can be critical for some deployments.

    Workaround: Obtain and install the NFS and Cinder backup debian packages.

    • apt-get install nfs-common
    • apt-get install cinder-backup
  • Kilo Heat templates incompatible with LBaaS v2.0.

    VMware Integrated OpenStack is based on the Kilo release of OpenStack and optionally supports LBaaS v2.0. However, Heat templates are not compatible with LBaaS v2.0 until the Mitaka release of OpenStack.

    Workaround: If you use Heat templates with VMware Integrated OpenStack 2.5 that require LBaaS resources, do not migrate to LBaaS v2.0. See the VMware Integrated OpenStack Administration Guide.

  • Interoperability with other VMware products with TLS v1.0 disabled.
    VMware Integrated OpenStack experiences interoperability issues with other VMware products when those products have disabled TLS v1.0 and SSL v3. Many clients are phasing out TLS v1.0 and SSL v3 because they are no longer considered secure by current revisions of the PCI Data Security Standard. Previous versions of VMware Integrated OpenStack disabled TLS v1.0 and SSL v3 on inbound public API connections. VMware Integrated OpenStack v2.5.1 and v3.0 fully interoperate with components that have disabled TLS v1.0 and SSL v3, including vSphere 6.0 update 2, NSX 6.2.4, and LDAP servers.

    Workaround: Disable TLS on the vCenter server where VMware Integrated OpenStack is running.

    1. Modify the /etc/vmware-rhttpproxy/config.xml file.
      <vmacore>
         <ssl>
            <doVersionCheck> false </doVersionCheck>
            <useCompression>true</useCompression>
            <libraryPath></libraryPath>
            <sslOptions> 117587968</sslOptions>
         </ssl>
      ...
    2. Modify the /etc/vmware-vpx/vpxd.cfg file.
      <vmacore>
         <cacheProperties>true</cacheProperties>
         <ssl>
            <useCompression>true</useCompression>
            <sslOptions> 117587968</sslOptions>
         </ssl>
      ...
    3. Restart the vpxd and rhttpproxy services on the vCenter server.
  • Metadata subnet ports not deleted.
    Stale metadata ports starting with IP 169.254.x.x persist in the Neutron database instead of being deleted.

    Workaround: Manually delete the ports using the neutron port-delete command.

  • Upgrade fails due to Neutron service failure.
    Observed when upgrading to 2.5 from 2.0.1 BUILD 4093159. In some cases, the Neutron service fails to start, causing the upgrade process to fail.

    NOTE: Check your current VMware Integrated OpenStack version. Version 2.0.1 BUILD 4093159 was a patch release provided only to specific customers.

    Workaround: Perform the following procedure before retrying the upgrade process.

    1. Log in to the VMware Integrated OpenStack management server,
    2. Download and install the 2.5.0 upgrade debian file.
      viopatch install -p vio-upgrade-2.5
    3. Download and add the 2.5.1 patch to the patch list.
      viopatch add -l vio-patch-251
    4. Install the 2.5.1 patch, assigning it a unique deployment name.
      viopatch install -p vio-patch-251 -v 2.5.1.4443700 -d unique_deployment_name --verbose
      Installing the patch does not affect the existing deployment nodes.
    5. Confirm the patch installation.

      root@vxlan-vm151:~# viopatch list
      Name               Version       Type   Partially-Installed   Installed
      ------------------ ------------- ------ --------------------- -----------
      vio-patch-251      2.5.1.4443700 infra  No                    Yes
      vio-upgrade-2.5    2.5.0.3955000 infra  No                    Yes
      vio-patch-201-hp-4 2.0.1.4093159 infra  No                    Yes

    6. Upgrade to 2.5.1 using the standard procedures. The process should succeed. See the VMware Integrated OpenStack Administration Guide.

  • Compute node data persists after cluster removal.
    In some cases, compute node-related info persists in the inventory file after the compute cluster is removed from the VMware Integrated OpenStack Manager interface. This issue can also disrupt the use of CLI commands.

    Workaround: Manually remove the persisting cluster configuration:

    1. Using SSH, log into the VMware Integrated OpenStack Manager console.
    2. Run viocli show -p to return the inventory file.
    3. Open the inventory file in an editor.
    4. In the inventory file, locate and delete the entry for the compute cluster to be removed.

      Compute cluster entries are located under the [compute] heading in the inventory file.

    5. Save the inventory file.

Updated Resolved Issues in 2.5 and 2.5.1

  • Host route injection doesn't work in VMware Integrated OpenStack 1.0.1

    Resolved in VMware Integrated OpenStack 2.5. Requires NSX 6.2.3.

  • Detaching volume fails with FileNotFoundException error.
    During volume detachment, the relocation of the shadow VM fails if storage DRS (SDRS) has moved the virtual disk. The following workaround works only if the volume does not contain snapshots.

    Resolved in VMware Integrated OpenStack 2.5.

  • Volume extend fails if a shadow VM has a disk chain.
    The extend volume feature is not supported when the volume contains one or more snapshots.

    Resolved in VMware Integrated OpenStack 2.5. Requires NSX 6.2.3.

  • Nova component is choked and becomes unreachable and times out.
    When Nova is choked for a long time, the master message queue becomes unreachable under the benchmark "NovaServers.boot_and_delete_servers".

    Resolved in VMware Integrated OpenStack 2.5.

  • Database node failure might block access to the dashboard for up to thirty minutes.
    If the master database node fails and restarts, users might not be able to log in to VMware Integrated OpenStack dashboard (Horizon) for up to thirty minutes. Observed in vCenter Server 5.5.

    Resolved in VMware Integrated OpenStack 2.5.

  • Nova boot creating a new volume from OVF fails.
    OVF is an unsupported format.

    Resolved in VMware Integrated OpenStack 2.5.

  • Need to run vioconfig start command if any OpenStack node is restarted
    If some of the node VMs in a cluster are restarted, you might need to restart the other nodes also. You can restart each node individually in the vSphere Web Client or by using the sudo vioconfig start command. If you cannot restart any of the nodes, ensure that the VMware Integrated OpenStack manager is running. Restart the manager first, then run sudo vioconfig start command to ensure all the OpenStack nodes start also.

    Resolved in VMware Integrated OpenStack 2.5.

  • Unable to reach node using SSH.
    A node might be unreachable externally through SSH, even though it might be reachable through SSH from the VIO management server. This has since been identified as due to network fluctuations outside the scope of vSphere and VMware Integrated OpenStack.

    Resolved in VMware Integrated OpenStack 2.5.

  • CPU fails to start due to certification failure.
    (SR 16194802807) When creating a Cinder volume, the Nova CPU fails because it cannot verify the certification.

    Resolved in VMware Integrated OpenStack 2.5.1.

  • Deployment fails with Failed to query IP address error.
    This failure occurs primarily in slow environments and might in some cases be related to Ubuntu bug 1326199. This issue is fast-tracked for resolution.

    Resolved in VMware Integrated OpenStack 2.5

  • Failure to recover multiple OpenStack VMs.
    After recovering multiple VMs, a rabbitMQ node doesn't recover and causes the config-mq failure.

    Resolved in VMware Integrated OpenStack 2.5.

  • Recovery of multiple VMs fails returns WSREP error.
    The command viocli recover -v -r RabbitMQ -n VIO-Controller-0 VIO-Memcache-0 VIO-LoadBalancer-0 VIO-DB-1 VIO-DB-2 fails with an error on the database nodes. This issue has been fast-tracked for resolution.

    Resolved in VMware Integrated OpenStack 2.5.

  • Glance API doesn't work after multi-VM recovery.
    Glance API doesn't work after multi-VM recovery and returns an Invalid OpenStack Identity credentials error.

    Resolved in VMware Integrated OpenStack 2.5.

  • Upgrade from VMware Integrated OpenStack 1.0.2 to 2.0 fails.
    There is a rare case in which the upgrade process fails with a service error. This is likely due to a Tomcat or Java loading class issue.

    Resolved in VMware Integrated OpenStack 2.5.

  • Summary page displays wrong version number after upgrade.
    After a patch upgrade, the Summary page in the VMware Integrated OpenStack manager might show the old version number.

    Resolved in VMware Integrated OpenStack 2.5.

  • Cinder backup-create defaults to NFS 4.1.
    When backing up block storage, OpenStack by default supports NFS 4.1. As a result, the cinder backup-create command fails for customers who use NFS 3.x.

    Resolved in VMware Integrated OpenStack 2.5. The procedures for creating cinder backups has been revised to accommodate different versions of NFS. See "Configure the Backup Service for Block Storage" in the VMware Integrated OpenStack Administrator Guide.

  • Firewall rules missing in NSX inventory.
    Firewall rules for tenant default security group might be missing in NSX inventory, resulting in lost connectivity between instances.

    Resolved in VMware Integrated OpenStack 2.5.

  • Stack deployment fails due to router issues.
    (SR 16149164306) Failure is caused because Neutron mistakes an exclusive router for a distributed one, even though the router is an exclusive one. Adding a router interface fails with flush error.

    Resolved in VMware Integrated OpenStack 2.5.1.

  • Neutron freezes, associated with Edges in PENDING_CREATE state.
    (SR 16197008407) Neutron locks up due to Edge lock requests failing, causing a 504 Gateway Timeout error for all users.

    Resolved in VMware Integrated OpenStack 2.5.1.

  • Neutron performance issues while trying to acquire Edge locks.
    (SR 16232743609) Neutron performance suffers and sometimes requires restarting due to problems associated with acquiring Edge locks.

    Resolved in VMware Integrated OpenStack 2.5.1.