Monitoring Memory with vSphere Web Client

An EM4J plug-in is available for the vSphere Web Client. The plug-in adds a Workloads tab that displays statistics for Java workloads on the virtual machine selected in the vSphere Web Client inventory tree. You can quickly verify that the virtual machine and JVM are configured correctly for EM4J and see detailed information about the JVM process and memory usage. Alerts warn of EM4J configuration problems and suggest best practices.

Setting Up the vSphere Web Client for EM4J

To monitor EM4J-enabled JVMs in the vSphere Web Client, the following components must be present and properly configured:

Download and Install the vSphere Web Client EM4J Plug-in

Procedure
  1. Browse to the VMware Downloads Web page at http://downloads.vmware.com.

  2. Under Application Platform, click VMware vFabric tc Server.

  3. Click the Driver & Tools tab.

  4. Click Download next to "VMware vFabric EM4J Console UI Plug-in 1.0".

  5. When prompted, log in with your VMware account and accept the license agreement.

  6. Save the file to your computer.

  7. Change to the vSphere Web Client plugin-packages directory. If you installed vSphere Web Client in the default location, use the following command:

    prompt> cd C:\Program Files\VMware\Infrastructure\vSphere Web Client\plugin-packages
  8. Enter the dir command to see if the em4j-client directory exists. If it exists, delete it and all of its contents:

    prompt> del /S/Q em4j-client
  9. Create the em4j-client directory:

    prompt> mkdir em4j-client
  10. In Windows Explorer, double-click the EM4J Console UI plug-in you downloaded and extract it into the em4j-client directory. You should have the following directory structure:

    vSphere Web Client/
        plugin-packages/
            em4j-client/
                help/
                legal/
                pluginPackage.xml
                plugins/
  11. Choose Start > Control Panel > Administrative Tools > Services.

  12. Right-click vSphere Web Client and choose Restart.

Setting the Required JVM System Properties

If you create a tc Runtime instance using the elastic-memory template, the system properties that enable EM4J are added to the java command line for you. To monitor the tc Runtime instances with the vSphere Web Client, an additional set of JMX properties is required. The CGC connects to the JVM using JMX, and it looks for these properties on the command line to establish the JMX connection. If the JMX properties are missing, the vSphere Web Client displays an alert to make you aware that the CGC could not establish the JMX connection.

The following table lists the JVM properties required to enable EM4J for the JVM and to allow the CGC to connect via JMX.

Table 3. EM4J Java System Properties

PropertyDescription
-javaagent=/path/to/balloon.jarPoints to the balloon.jar EM4J Java library, which can be found in the lib directory of the tc Runtime instance. This property is added for you when you create a tc Runtime instance using the elastic-memory template.
-agentpath=/path/to/native/libraryPoints to the native EM4J library, which is in an architecture-specific subdirectory of the lib directory in the tc Runtime instance directory. This property is added for you when you create a tc Runtime instance using the elastic-memory template.
-Dcom.sun.management.jmxremote=trueEnables JMX remote agent and local monitoring. You must manually add this property to the tc Runtime instance.
-Dcom.sun.management.jmxremote.port=portnoCreates a JMX connector listening on the specified port. You must manually add this property to the tc Runtime instance.
-Dcom.sun.management.jmxremote.authenticate=falseDisables authentication for JMX. You must manually add this property to the tc Runtime instance.
-Dcom.sun.management.jmxremote.ssl=falseDisables JMX monitoring via SSL. You must manually add this property to the tc Runtime instance.

If you use the tcruntime-instance command with the elastic-memory template to create the tc Runtime instance, the -javaagent and -agentpath system properties are set up and the required Java and native libraries are copied into the tc Runtime instance.

For example, this command, executed in the tc Server installation directory, creates an EM4J-enabled tc Runtime instance named myInstance:

prompt$ ./tcruntime-instance.sh create myInstance -t elastic-memory

Once the tc Runtime instance is created, edit the $CATALINA_HOME/bin/setenv.sh file and add the JMX system properties to the JVM_OPTS variable:

Note

Lines are wrapped for readability.

JVM_OPTS="-Dcom.sun.management.jmxremote=true 
          -Dcom.sun.management.jmxremote.port=6969 
          -Dcom.sun.management.jmxremote.authenticate=false 
          -Dcom.sun.management.jmxremote.ssl=false"

Running the Console Guest Collector

To view real, current data in the vSphere Web Client, you must configure cron to run the Console Guest Collector (CGC) periodically on each virtual machine with Java workloads. The CGC gathers information about the workloads and saves it in an area of the virtual machine where the vSphere Web Client can retrieve it.

JMX properties must be specified on the java command line for each JVM to allow the CGC to find the information it requires to connect to the JVM. The CGC reads the process table to find Java processes and then extracts the JMX connection properties from the command line. See Setting the Required JVM System Properties.

Note

A cgc.sh script exists in the templates/elastic-memory/bin directory of your tc Server installation and in the $CATALINA_HOME/bin directory of each tc Server instance you create using the elastic-memory template. You can run cgc.sh from any of these locations, but you should only run one instance per virtual machine.

You must have OS-level permission to edit the crontab.

Procedure
  1. Log in to the guest operating system as root or as a user with sudo permission.

  2. Edit the crontab table for the user who executes tc Runtime instances, for example, the tc-server user.

    sudo crontab -u tc-server -e
  3. Add a line to execute cgc.sh. This example uses the cgc.sh located in the templates/elastic-memory/bin directory in the tc Server installation directory.

    Note

    Lines are wrapped for readability. The entire crontab entry is a single line.

    */5 * * * * /opt/vmware/vfabric-tc-server-standard-2.6.0.RELEASE/templates/elastic-memory/bin/cgc.sh > 
    /dev/null 2>&1

    This crontab entry executes cgc.sh once every 5 minutes. See the man page for cron to modify the crontab entry to run the CGC at different time intervals.

  4. Save the crontab file.

The cgc.sh script executes a Java class that collects information about each of the JVMs running on the virtual machine and saves it in an area of the virtual machine where the EM4J plug-in for the vSphere Web Client can retrieve it. The vSphere Web Client displays information collected by the most recent CGC execution.

Disabling the vSphere Web Client EM4J Plug-in

You can disable the EM4J plug-in in the vSphere Web Client Plug-in Management console. The plug-in remains installed, but is inoperative. You can re-enable the plug-in later using a similar procedure.

Procedure
  1. From the Applications menu, choose System Administration > Plug-in Management.

  2. Right-click Elastic Memory for Java Client and choose Other > Disable from the context menu.

  3. Click Yes. A Reload vSphere Web Client dialog box appears.

  4. Click Yes.

Viewing Java Workloads in the vSphere Web Client

When the vSphere Web Client EM4J plug-in is installed, select any virtual machine in the inventory tree and then click the Workloads tab.

If the virtual machine has no JVMs, or if EM4J is not properly configured in the virtual machine, the list of workloads is empty and alert messages report why no workloads were discovered.

When JVMs are discovered on the virtual machine, they appear in the workloads list.

The following table describes the data presented in the workloads list.

Table 4. Workloads List

ColumnDescription
NameThe name of the workload, normally java.
StatusThe workload status: Normal, Information, or Warning. It is a rollup value, indicating the highest severity of any alerts that exist for the workload.
EM4J Agent EnabledA checkbox indicating if the EM4J agent is enabled. The EM4J agent is enabled when the ESXi version is supported for EM4J, VMware tools are installed, and the sched.mem.pshare.guestHintsSyncEnable configuration parameter is set to true.
Main ClassThe name of the main Java class the JVM is executing.
Virtual MachineThe name of the virtual machine where the JVM is executing.
HostThe name of the ESXi host executing the virtual machine.

When a JVM is selected in the workloads list, you can select from three tabs to display information about the JVM:

Summary Tab

The Workload Summary tab shows information about the workload selected in the Workloads list.

Status

The Status section shows information about the state of EM4J for workload.

Table 5. Attributes in the Status Section

AttributeDescription
OverallThe overall status for EM4J, either Normal, Information, or Warning.
EM4J Agent EnabledA checkbox indicating if the EM4J agent is enabled. The EM4J agent is enabled when the ESXi version is supported for EM4J, VMware tools are installed, and the sched.mem.pshare.guestHintsSyncEnable configuration parameter is set to true.

Annotations

The Annotations section displays descriptive information associated with the workload. If EM4J is able to determine the name of the application the JVM is running, it is displayed here.

JVM Details

The JVM Details section displays information about the Java virtual machine.

Table 6. Attributes in the JVM Details Section

AttributeDescription
PIDThe operating system process ID for the executing JVM process.
Main ClassThe name of the main Java class the JVM executed on startup.
Large Pages Supported by Guest OSA checkbox indicating if /proc/sys/vm/nr_hugepages exists with non-zero contents, which signals that the guest OS is configured to use large memory pages. Using large memory pages is a best practice for running Java on a virtual machine, but is not a requirement to use EM4J.
Number of Large PagesThe number of large pages configured in the Guest OS; the contents of /proc/sys/vm/nr_hugepages.
JVM Using Large PagesA checkbox indicating if the JVM is using large memory pages.JVM large page support is enabled with the -XX:+UseLargePages Java command line option.

Related Items

The Related Items section lists information about the location of the virtual machine in the vSphere Web Client inventory.

Table 7. Attributes in the Related Items Section

AttributeDescription
Virtual MachineThe name of the virtual machine where the workload is executing.
HostThe host name or IP number of the ESXi host.
Resource PoolThe name of the resource pool the virtual machine belongs to.

Alerts Tab

The Alerts tab displays a list of alerts that have been activated in response to the state of a Java workload. Alerts are messages that report real or potential configuration problems that prevent EM4J from operating normally.

EM4J alerts have a severity of Information, Warning, or Java Best Practice. An alert with a severity of Information reports a non-error condition that prevents EM4J from running. For example, if there are no JVMs running on the virtual machine, an alert with Information severity reports the fact.

An alert with Warning severity reports an error condition that prevents EM4J from operating normally. For example, if the Console Guest Collector on a virtual machine is unable to connect to the JMX port of a JVM, an alert with Warning severity is issued.

Java Best Practice alerts are suggestions to help improve performance when running Java on a virtual machine, for example, using large memory pages when the OS supports them.

The Status displayed on the Workloads list is the most severe alert level for the workload, or Normal if there are no alerts. When determining the severity, Java Best Practice alerts are treated as Information alerts.

A list of the potential alerts appears in the Elastic Memory for Java Client Plug-in Online Help topic for the Alerts tab.

Resource Management Tab

The Resource Management tab displays information about memory usage in JVMs and the virtual machine, and JVM garbage collections. The EM4J memory statistics displayed on the Resource Management tab are a snapshot of memory utilization and garbage collection when the Console Guest Collector last ran on the guest OS.

Note

The statistics shown in the Guest Memory section are updated more frequently than the JVM Heap Memory statistics and provide the most up-to-date and accurate measure of ballooned memory size for the virtual machine.

The JVM Heap memory statistics include details reported by all JVMs running when the CGC executed last. When a JVM exits, its balloon memory can persist in the ESXi-managed balloon for the virtual machine, although it is no longer associated with a JVM. For this reason, the total balloon size reported for all EM4J-enabled JVMs may be less than the actual balloon size.

Heap Memory: Selected JVM

This section includes a stacked bar graph depicting the composition of heap memory in the JVM selected in the Workloads list.

Table 8. JVM Heap Memory Statistics

AttributeDescription
Application Heap MemoryAmount of heap memory currently used, excluding EM4J ballooned and shareable memory.
Ballooned Heap Memory (EM4J Balloon)Amount of the EM4J memory reclaimed by ESXi.
EM4J Shareable Heap MemoryAmount of heap memory objects owned by EM4J that are not currently ballooned, but can still be reclaimed by ESXi as zero-shared memory.
Free Heap MemoryAmount of free heap memory.
Committed Heap MemoryAmount of heap memory currently allocated to the JVM. It varies as the JVM requests additional memory or releases memory to the operating system. Committed Heap Memory is the sum of Application Heap Memory, Ballooned Heap Memory, EM4J Shareable Heap Memory, and Free Heap Memory.

The portion of heap memory that is in use contains objects stored by applications running on the JVM (Application Heap Memory) as well as objects created by EM4J. Of the memory controlled by EM4J, some may be reclaimed by ESXi to satisfy the balloon target for the virtual machine (Ballooned Heap Memory) and the remainder is zero-filled, making it available to ESXi to reclaim with transparent page sharing (EM4J Shareable Memory).

Heap Memory: All EM4J-Enabled JVMs

This section includes a stacked bar graph depicting the composition of heap memory for all EM4J-enabled JVMs running on the virtual machine.

Table 9. JVM Heap Memory Statistics

AttributeDescription
Total Application Heap MemoryTotal amount of heap memory used by all JVMs on the virtual machine, excluding EM4J ballooned and shareable memory.
Total Ballooned Heap Memory (EM4J Balloon)Total amount of heap memory reclaimed by all EM4J balloons on the virtual machine.
Total EM4J Shareable Heap MemoryTotal amount of heap memory objects owned by EM4J that are not currently ballooned, but that could be reclaimed by ESXi as zero-shared memory.
Total Free Heap MemoryTotal amount of free heap memory for all JVMs on the virtual machine.
Total Committed Heap MemoryTotal amount of committed heap memory for all JVMs on the virtual machine.

Guest Memory

The Guest Memory section displays information about guest memory usage. The statistics in this section are live, unlike other sections displayed on this tab, which are from the most recent CGC execution.

Table 10. Guest Memory Statistics

AttributeDescription
Active Guest MemoryAmount of memory recently accessed.
Private Guest MemoryAmount of memory backed by host memory and not being shared.
Shared Guest MemoryAmount of the virtual machine's memory being shared.
Ballooned Guest MemoryAmount of the virtual machine's memory reclaimed by ballooning.
Compressed Guest MemoryAmount of the virtual machine's memory in compression cache.
Swapped Guest MemoryAmount of the virtual machine's memory reclaimed by swapping.
Unaccessed Guest MemoryAmount of the virtual machine's memory never referenced by the guest.

Garbage Collectors

The Garbage Collectors section displays a list of the JVM's garbage collectors.

Table 11. Garbage Collectors Statistics

AttributeDescription
NameName of the garbage collector.
Collection CountCumulative number of garbage collections.
Collection Time (ms)Cumulative execution time of the garbage collector, in milliseconds.

Initial Memory Configuration

The Initial Memory Configuration section describes the initial JVM memory layout as configured on the java command line.

Table 12. Initial Memory Configuration Statistics

AttributeDescription
Minimum Heap SizeMinimum heap size, specified with the -Xms system property.
Maximum Heap SizeMaximum heap size, specified with the -Xmx JVM option.
Maximum PermGen SizeMaximum size of the permanent generation, specified with the -XX:MaxPermSizeJVM option.
Thread Stack SizeSize of the stack for each thread, specified with the -Xss JVM option.
Using Large PagesChecked if the JVM is using large memory pages. Specified with the -XX:+UseLargePages JVM option. Large memory support must be configured in the OS to enable this option.