VMware Integrated OpenStack 2.0 Release Notes

VMware Integrated OpenStack 2.0 | 10 SEPTEMBER 2015 | BUILD 3037963

What's in the Release Notes

The release notes cover the following topics:

About VMware Integrated OpenStack

VMware Integrated OpenStack greatly simplifies deploying an OpenStack cloud infrastructure by streamlining the integration process. VMware Integrated OpenStack delivers out-of-the-box OpenStack functionality and an easy configuration workflow through a deployment manager vApp that runs directly in vCenter.

All information about this release is highly confidential and subject to the non-disclosure terms and conditions in the Master Software Beta Test Agreement that your company has already agreed to.

New Features in this Release

VMware Integrated OpenStack enables the rapid deployment of OpenStack on a VMware vSphere virtual platform. This release provides the following new features and enhancements.

  • Seamless Upgrade to Kilo release of OpenStack. VMware Integrated OpenStack 1.0.x was based on the Icehouse release. By upgrading to 2.0, you skip the Juno release. After upgrading, you have the option to roll back to 1.0.x.

  • Additional Language Support. VMware Integrated OpenStack 2.0 is available in six additional Languages. French, German, Japanese, Korean, Simplified Chinese, and Traditional Chinese.

  • Improved and Reliable vApp Registration. The process for registering the VMware Integrated OpenStack plugin has been streamlined.

  • LBaaS Integration. VMware Integrated OpenStack 2.0 supports LBaaS 1.0 (Load-Balancing-as-a-Service), which is an advanced Neutron service in OpenStack. LBaaS was first introduced in the Juno release of OpenStack.

  • Ceilometer Integration. This is an OpenStack service that collects measurements of the utilization of the resources that comprise your VMware Integrated OpenStack deployment. Ceilometer persists the collected data for subsequent retrieval and analysis in a MongoDB backend. Note: As an OpenStack component, Ceilometer is in technology preview at this time.

  • Heat Auto Scaling. This is an advanced Heat service that enables users to set up metrics that will scale up or down application components. This enables development teams to address unpredictable changes in demand for the app services. Ceilometer provides the alarms and triggers, Heat orchestrates the creation (or deletion) of scale-out components, and LBaaS provides load balancing for the scale-out components.

  • Advanced vSphere Integration. VMware Integrated OpenStack 2.0 exposes VMware vSphere© Windows Guest Customization. VMware admins can specify various attributes such as ability to generate new SIDs, assign admin passwords for the VM, manage compute names, and so on. There is also added support for more granular placement of VMs by leveraging vSphere features such as affinity and anti-affinity settings.

  • Image Metadata Introspection and VMware-specific image metadata (vmware_adaptertype, vmware_disktype, and so on) can be introspected and saved with the image. Introspection is supported on sparse and streamOptimized VMDK images and OVAs.

  • Automatic Image Conversion. VMware Integrated OpenStack 2.0 enables you to import and convert the following types of disk images: QCOW2, RAW, VHD and VDI. The images are automatically converted to a suitable VMDK format.

  • Backup and Restore. VMware Integrated OpenStack 2.0 enables you to backup and restore your OpenStack services and configuration data through the VMware Integrated OpenStack manager.

  • Improved Recovery VMware Integrated OpenStack 2.0 includes new features to enable you to recover your control plane from failure.

Updated Compatibility

The VMware Product Interoperability Matrix provides details about the compatibility of the current version of VMware Integrated OpenStack with VMware vSphere components, including ESXi, VMware vCenter Server, the vSphere Web Client, and optional VMware products. Check the VMware Product Interoperability Matrix also for information about supported management and backup agents before you install VMware Integrated OpenStack or other VMware products.

Upgrading to VMware Integrated OpenStack 2.0

You perform the upgrade procedure directly in the VMware Integrated OpenStack manager. Because vSphere will have to accommodate the existing deployment and the upgrade files, the upgrade process requires you to temporarily double the datastore resources.


  • Double the datastore resources dedicated to your current VMware Integrated OpenStack deployment.
  • As a precaution, back up your current deployment. See the VMware Integrated OpenStack Administration Guide for details.
  • Preserve your current VMware Integrated OpenStack deployment configuration by exporting it as a template.

To upgrade to VMware Integrated OpenStack 2.0:

  1. In the vSphere Web Client, select Home > Inventories, and click the VMware Integrated OpenStack icon.

  2. Click the Manage tab and click the Upgrades tab.

  3. Click the Upgrade Operation tab.

    This tab lists the current VMware Integrated OpenStack deployment.

  4. Right-click the deployment name and choose Upgrade from the pop-up menu.

  5. Enter the name for the new deployment, and click Next.

  6. Configure the public and private VIP settings for the load balancer service.

    • Public Virtual IP This value should be on the same subnet as the OpenStack API Access network, and should be outside the IP range specified for the OpenStack API Access network.

    • Private Virtual IP This IP connects the load balancer interface with the Management network.

  7. Click Next.

  8. Review the upgrade configuration, and click Finish.

Open Source Components for VMware Integrated OpenStack 2.0

The copyright statements and licenses applicable to the open source software components distributed in VMware Integrated OpenStack 2.0 are available on the Open Source tab of the product download page. You can also download the source files for any GPL, LGPL, or other similar licenses that require the source code or modifications to source code to be made available for the most recent available release of VMware Integrated OpenStack.

Updated Updated Known Issues

VMware Integrated OpenStack 2.0 has the following known issues. If you encounter an issue that is not in this known issues list, search the VMware Knowledge Base, or let us know by contacting VMware Technical Support.

  • Unable to modify syslog setting post deployment in VIO Manager interface

    After deploying VIO, you cannot modify the syslog server configuration using the setting in the VIO Manager interface (VMware Integrated OpenStack > Management Server > Edit Settings > vApp Options).

    Workaround: Modify the configuration here: VMware Integrated OpenStack > OpenStack Cluster > Manage > LogInsight.

  • Dashboard might show a Volume as attached even if it failed to attach.
    This is a known OpenStack issue, first reported in the Icehouse release.

  • Long image upload times cause NotAuthenticated failure.
    This is a known OpenStack issue (https://bugs.launchpad.net/glance/+bug/1371121), first reported in the Icehouse release.

  • Detaching volume fails with FileNotFoundException error.
    During volume detachment, the relocation of the shadow VM fails if storage DRS (SDRS) has moved the virtual disk. The following workaround works only if the volume does not contain snapshots.

    Workaround: The following procedure works only if the volume does not contain snapshots.

    1. In detach_volume, reconfigure the shadow VM to remove its VMDK.

      NOTE: Do not set the option to destroy files.

      This will remove the disk device even if the backing files do not exist and the VM is now ready for relocate

    2. Destroy the volume's VMDK.

      NOTE: Ignore the filenotfound exception. This implies that SDRS has moved the VMDK instead of copying; because there are no references to the VMDK, you must delete it explicitly.

    3. Relocate the shadow VM to the datastore of the instance.

    4. Reconfigure the shadow VM to attach the latest copy of volume's VMDK.

    5. Reconfigure the instance to remove volume's VMDK.

  • Updated Host route injection doesn't work in VMware Integrated OpenStack 1.0.1
    This issue has been fast-tracked for resolution.

  • Volume extend fails if a shadow VM has a disk chain.
    The extend volume feature is not supported when the volume contains one or more snapshots.

  • OpenStack dashboard does not immediately reflect volume extension.
    If you extend the size of a volume, the dashboard might still display the pre-extended size. Eventually, the correct size appears.

  • Nova component is choked and becomes unreachable and times out.
    When Nova is choked for a long time, the master message queue becomes unreachable under the benchmark "NovaServers.boot_and_delete_servers".

    Workaround: Restart the nova-conductor on both controller nodes, and restart nova-compute on the Compute node.

  • OpenStack Management Service does not restart after vSphere HA event.
    If vSphere experiences an HA event (for example, the ESXi host hosting the OpenStack Management Server), when vSphere HA restarts, the OpenStack Management Service remains is not restarted. Observed in vCenter Server 5.5.

    Workaround: If a vSphere HA event occurs that affect the OpenStack deployment, ensure that the OpenStack Management Service has restarted. Restart manually if necessary.

  • Database node failure might block access to the dashboard for up to thirty minutes.
    If the master database node fails and restarts, users may not be able to log in to VMware Integrated OpenStack dashboard (Horizon) for up to thirty minutes. Observed in vCenter Server 5.5.

  • Installer gives priority to local storage by default.
    When provisioning datastores for the three database VMs, the VMware Integrated OpenStack installer automatically gives priority to local storage to improve I/O performance. For resiliency, users might prefer shared storage, but the installer does not make it clear how to modify this setting.

    Workaround: Before you complete the installation process, the VMware Integrated OpenStack installer allows you to review and modify the configuration. You can use this opportunity to change the datastore configuration for the three database VMs.

  • Special characters in datastore names not supported by Glance (Image Service).
    If a datastore name has non-alphanumeric characters like colons, ampersands, or commas, the datastore cannot be added to the Glance service. Specifically, the following characters are not permitted in Glance datastore names because their use is reserved for other purposes and therefore can interfere with the configuration: : , / $ (colon, comma, forward slash, dollar).This issue has been fast-tracked for resolution.

  • Nova boot creating a new volume from OVF fails.
    OVF is an unsupported format.

    Workaround: Convert the OVF to an OVA using the appropriate VMware tool or use another supported format.

  • VMware Integrated OpenStack dashboard (Horizon) throws a 504 Gateway Timeout.
    After approximately 1500 networks have been configured, clicking Networks in the dashboard navigation panel might result in a 504 error.

  • If either controller VM reboots, high availability might be compromised.
    When a controller fails, the other controller continues to provide services. However, when the initial controller reboots, it might no longer provides services, and thus is not available if the other controller also fails. This issue has been fast-tracked for resolution.

    Workaround: If a controller fails and HA is invoked, review your deployment to ensure that both controllers are providing services after the failed controller reboots.

  • vSphere Datacenter Names with Commas Prevent Deployment.
    If the VMware vSphere Datacenter name has a comma in it (for example: "Palo Alto, CA"), you will be unable to deploy VMware Integrated OpenStack. The deployment wizard displays an invalid parameter error.

    Workaround: If possible modify datacenter names to be used for VMware Integrated OpenStack deployments to omit the "," character.

  • Need to run vioconfig start command if any OpenStack node is restarted
    If some of the node VMs in a cluster are restarted, you might need to restart the other nodes also. You can restart each node individually in the vSphere Web Client or by using the sudo vioconfig start command. If you cannot restart any of the nodes, ensure that the VMware Integrated OpenStack manager is running. Restart the manager first, then run sudo vioconfig start command to ensure all the OpenStack nodes start also.

  • "Nova boot image" fails intermittently.
    If the OpenStack Manager IP address is changed (for example, by modifying the FQDN setting), thus causing the OpenStack VMs to fall out of synchronization with the OpenStack Management Service. This bug is fast-tracked for correction.

    Workaround: Manually modify the OpenStack Manager IP address.

    1. From the vSphere Web interface, choose VMware Integrated OpenStack vApp Get Started > Edit Settings... > Networking Properties.

    2. Change the Management Server Network IP Address to the new one.

    3. Power off and restart the VMware Integrated OpenStack vApp.

    4. Using SSH, log in to Management Server console.

    5. Navigate to the /etc/network/interfaces file and edit the IP address setting to the new one.

    6. Restart the OpenStack VMs.

  • Unable to reach node using SSH.
    A node might be unreachable externally through SSH, even though it may reachable through SSH from the VIO management server. This has since been identified as due to network fluctuations outside the scope of vSphere and VMware Integrated OpenStack.

    Workaround: You can discover the specific cause by enabling debugging for the jarvis service.

    1. Log in to the management server. and edit file /etc/jarvis/jarvis.conf and change line "verbose=False" to "verbose=True"

    2. Edit the file /etc/jarvis/jarvis.conf by changing "verbose=False" to "verbose=True".

    3. Save and close the file.

    4. Restart jarvis: sudo service jarvis restart.

    5. Run the deployment again.

      NOTE: Verbose logging output will quickly fill the ansible log. Reset to False after you have debugged the issue.

  • Metadata service is not accessible on subnets created with the no-gateway option.
    In 2.0, autoconfiguration is turned off for Edge VMs. When applicable, DHCP sets the gateway and metadata is served through this gateway Edge. As a result, when a subnet is created with the no-gateway option, there is no router Edge to capture the metadata traffic.

    Workaround:For networks with the no-gateway option, configure a route for to forward traffic to DHCP Edge IP.

  • Failure to recover multiple OpenStack VMs.
    After recovering multiple VMs, a rabbitMQ node doesn't recover and causes the config-mq failure.

    Workaround: Recover the entire VM tier:

    viocli recover -v -r RabbitMQ -n VIO-Controller-0 VIO-ComputeDriver-0 VIO-Memcache-1 VIO-LoadBalancer-1 VIO-DB-0

  • Deployment fails with Failed to query IP address error.
    This failure occurs primarily in slow environments and might in some cases be related to Ubuntu bug 1326199. This issue is fast-tracked for resolution.

    Workaround: You work around this issue by modifying the rootdelay=180 parameter setting in the /boot/grub/menu.lst file.

    1. Remove the snapshot of the VMware Integrated OpenStack template VM.
    2. Start the VMware Integrated OpenStack template VM.
    3. Using SSH, log in to the template VM CLI.
    4. Open the /boot/grub/menu.lst file in a text editor.
    5. Add the rootdelay=180 parameter to line 135.
    6. Save the /boot/grub/menu.lst file.
    7. Clear all contents from /etc/network/interfaces directory.
    8. Stop the VMware Integrated OpenStack template VM.
    9. In vCenter, delete the failed deployment.
    10. Redeploy VMware Integrated OpenStack.
  • Post upgrade: duplicate image caches
    When you upgrade from VMware Integrated OpenStack 1.0 to 2.0, the existing image caches are duplicated. The 1.0 images caches are required to support all instances created in 1.0; instances created in 2.0 use the new image cache.

    Workaround: Do not delete the 1.0 image caches if you still have instances created in 1.0, or the instances will fail. When you have no more 1.0 instances, you can delete the 1.0 image caches.

  • Horizon dashboard shows error after switching projects as admin user
    If you are logged into Horizon as the administrative user and try to switch between projects using the dropdown menu in Horizon, the dashboard might begin returning errors. This is caused by an issue in OpenStack.

    Workaround: Log out of the Horizon dashboard and log back in to restore the dashboard.

  • Member statuses not regularly updated only while reading pool stats.
    Observed in deployments with NSX 6.2 and VMware Integrated OpenStack 2.0 with LBaaS enabled. Load balancing member node statuses are updated only when pool statuses are fetched. They should be updated periodically and independently of pool statuses.

  • Problems with heat template creating ScalingPolicy.
    Ceilometer Heat templates cannot create ScalingPolicy as a tenant member, only as a tenant admin. This appears to be a permissions issue. This issue is fast-tracked for resolution.

    Workaround: If you do not use LDAP in your deployment, you can work around this issue by manually modifying the Keystone configuration to give the user and tenant owner privileges.

    keystone user-role-add --user USER_NAME --tenant TENANT_NAME --role heat_stack_owner

    Note: You must restart the VMware Integrated OpenStack template for the modified setting to take effect.

  • WaitConditionHandle not working with current heat.conf.
    Using OS::Heat::WaitConditionHandle in a Heat Orchestration Template (HOT) template might generate the following error:

    Resource Create Failed: Error: Resources.Mysql.Resources.Wait Handle: Cannot Get Stack Domain User Token, No Stack Domain Id Configured, Please Fix Your Heat.Conf

    This issue has been fast-tracked for resolution.

  • Recovery of multiple VMs faile returns WSREP error.
    The command viocli recover -v -r RabbitMQ -n VIO-Controller-0 VIO-Memcache-0 VIO-LoadBalancer-0 VIO-DB-1 VIO-DB-2 fails with an error on the database nodes. This issue has been fast-tracked for resolution.

    Workaround: Use the failure recovery procedures to recover the database nodes and restart the VMware Integrated OpenStack deployment. See the VMware Integrated OpenStack Administration Guide for details.

  • Upgrade to 2.0 Fails with "No qualified ESXi Host for Controller-1" error
    This error is caused because user customizations to the omjs.properties file are not passed to 2.0 during the upgrade process.

    Workaround: Reapply your customizations to the 2.0 omjs.properties file and restart the VMware Integrated OpenStack manager.

  • Glance API doesn't work after multi-VM recovery.
    Glance API doesn't work after multi-VM recovery and returns an Invalid OpenStack Identity credentials error.

    Workaround: You can work around this issue by stopping and starting your deployment.

    viocli deployment stop
    viocli deployment start

  • Reverting from 2.0 to 1.0.x results in error alert.
    If you revert to 1.0.x, VMware Integrated OpenStack might show an authentication error alert.

    Workaround: Close the alert and VMware Integrated OpenStack manager, and restart the vSphere Web client service. When you restart the VMware Integrated OpenStack manager, the VMware Integrated OpenStack service will restart without the error.

  • Snapshot upload fails due to timeout error.
    Snapshot upload fails due to HTTP read timeout in vCenter.

    Workaround: You can prevent this error by modifying the readTimeoutMs setting in the vpxd.cfg file.

    1. As a precaution, make a backup copy of the vpxd.cfg file.

    2. Modify the readTimeoutMs setting to 600000.

    3. Stop and restart vCenter.

  • Upgrade from VMware Integrated OpenStack 1.0.2 to 2.0 fails.
    There is a rare case in which the upgrade process fails with a service error. This is likely due to a Tomcat or Java loading class issue.

    Workaround: Restart the VMware Integrated OpenStack manager and try again. The procedure should succeed.

Resolved Issues

  • CLI actions for stop/start not reflected in vSphere Web Client.
    Fixed in VMware Integrated OpenStack 2.0.

  • Edge VMs require shared datastores.
    Fixed in VMware Integrated OpenStack 2.0.

  • Without NAT, firewall rules prevent external-to-internal communication.
    Fixed in VMware Integrated OpenStack 2.0.

  • Adding new tenant logical router to shared Edge breaks static route.
    Fixed in VMware Integrated OpenStack 2.0.

  • Image operating system type mismatch.
    Fixed in VMware Integrated OpenStack 2.0.

  • Cluster with NSX-V fails to deploy due to "$" character in password.
    Fixed in VMware Integrated OpenStack 2.0.