Monitoring Memory with vSphere Web Client

An EM4J plug-in is available for the vSphere Web Client. The plug-in adds a Workloads tab that displays statistics for Java workloads on the virtual machine selected in the vSphere Web Client inventory tree. You can quickly verify that the virtual machine and JVM are configured correctly for EM4J and see detailed information about the JVM process and memory usage. Alerts warn of EM4J configuration problems and suggest best practices.

Setting Up the vSphere Web Client for EM4J

To monitor EM4J-enabled JVMs in the vSphere Web Client, the following components must be present and properly configured:

vCenter Server. See Installing vCenter Server.
vSphere Web Client. See lnstall and Start the vSphere Web Client.
vSphere Web Client EM4J plug-in. See Download and Install the vSphere Web Client EM4J Plug-in.
Virtual machines, VM Version 7 or higher, with VMware Tools installed and sched.mem.pshare.guestHintsSyncEnable configuration property set to true. See Enable EM4J in the Virtual Machine.
tc Runtime instances started with the required EM4J and JMX properties specified on the java command line. See Setting the Required JVM Properties.
CGC set up in cron. See Running the Console Guest Collector.

Download and Install the vSphere Web Client EM4J Plug-in

Procedure

Browse to the VMware Downloads Web page at http://downloads.vmware.com.
Under Application Platform, click VMware vFabric tc Server.
Click the Driver & Tools tab.
Click Download next to "VMware vFabric EM4J Console UI Plug-in 1.0".
When prompted, log in with your VMware account and accept the license agreement.
Save the file to your computer.
Change to the vSphere Web Client plugin-packages directory. If you installed vSphere Web Client in the default location, use the following command:
```
prompt> cd C:\Program Files\VMware\Infrastructure\vSphere Web Client\plugin-packages
```
Enter the dir command to see if the em4j-client directory exists. If it exists, delete it and all of its contents:
```
prompt> del /S/Q em4j-client
```
Create the em4j-client directory:
```
prompt> mkdir em4j-client
```
In Windows Explorer, double-click the EM4J Console UI plug-in you downloaded and extract it into the em4j-client directory. You should have the following directory structure:
```
vSphere Web Client/
    plugin-packages/
        em4j-client/
            help/
            legal/
            pluginPackage.xml
            plugins/
```
Choose Start > Control Panel > Administrative Tools > Services.
Right-click vSphere Web Client and choose Restart.

Setting the Required JVM System Properties

If you create a tc Runtime instance using the elastic-memory template, the system properties that enable EM4J are added to the java command line for you. To monitor the tc Runtime instances with the vSphere Web Client, an additional set of JMX properties is required. The CGC connects to the JVM using JMX, and it looks for these properties on the command line to establish the JMX connection. If the JMX properties are missing, the vSphere Web Client displays an alert to make you aware that the CGC could not establish the JMX connection.

The following table lists the JVM properties required to enable EM4J for the JVM and to allow the CGC to connect via JMX.

Table 3. EM4J Java System Properties

Property	Description
`-javaagent=/path/to/balloon.jar`	Points to the balloon.jar EM4J Java library, which can be found in the lib directory of the tc Runtime instance. This property is added for you when you create a tc Runtime instance using the `elastic-memory` template.
`-agentpath=/path/to/native/library`	Points to the native EM4J library, which is in an architecture-specific subdirectory of the `lib` directory in the tc Runtime instance directory. This property is added for you when you create a tc Runtime instance using the `elastic-memory` template.
`-Dcom.sun.management.jmxremote=true`	Enables JMX remote agent and local monitoring. You must manually add this property to the tc Runtime instance.
`-Dcom.sun.management.jmxremote.port=portno`	Creates a JMX connector listening on the specified port. You must manually add this property to the tc Runtime instance.
`-Dcom.sun.management.jmxremote.authenticate=false`	Disables authentication for JMX. You must manually add this property to the tc Runtime instance.
`-Dcom.sun.management.jmxremote.ssl=false`	Disables JMX monitoring via SSL. You must manually add this property to the tc Runtime instance.

If you use the tcruntime-instance command with the elastic-memory template to create the tc Runtime instance, the -javaagent and -agentpath system properties are set up and the required Java and native libraries are copied into the tc Runtime instance.

For example, this command, executed in the tc Server installation directory, creates an EM4J-enabled tc Runtime instance named myInstance:

prompt$ ./tcruntime-instance.sh create myInstance -t elastic-memory

Once the tc Runtime instance is created, edit the $CATALINA_HOME/bin/setenv.sh file and add the JMX system properties to the JVM_OPTS variable:

Note

Lines are wrapped for readability.

JVM_OPTS="-Dcom.sun.management.jmxremote=true 
          -Dcom.sun.management.jmxremote.port=6969 
          -Dcom.sun.management.jmxremote.authenticate=false 
          -Dcom.sun.management.jmxremote.ssl=false"

Running the Console Guest Collector

To view real, current data in the vSphere Web Client, you must configure cron to run the Console Guest Collector (CGC) periodically on each virtual machine with Java workloads. The CGC gathers information about the workloads and saves it in an area of the virtual machine where the vSphere Web Client can retrieve it.

JMX properties must be specified on the java command line for each JVM to allow the CGC to find the information it requires to connect to the JVM. The CGC reads the process table to find Java processes and then extracts the JMX connection properties from the command line. See Setting the Required JVM System Properties.

Note

A cgc.sh script exists in the templates/elastic-memory/bin directory of your tc Server installation and in the $CATALINA_HOME/bin directory of each tc Server instance you create using the elastic-memory template. You can run cgc.sh from any of these locations, but you should only run one instance per virtual machine.

You must have OS-level permission to edit the crontab.

Procedure

Log in to the guest operating system as root or as a user with sudo permission.
Edit the crontab table for the user who executes tc Runtime instances, for example, the tc-server user.
```
sudo crontab -u tc-server -e
```
Add a line to execute cgc.sh. This example uses the cgc.sh located in the templates/elastic-memory/bin directory in the tc Server installation directory.
Note
Lines are wrapped for readability. The entire crontab entry is a single line.
```
*/5 * * * * /opt/vmware/vfabric-tc-server-standard-2.6.0.RELEASE/templates/elastic-memory/bin/cgc.sh > 
/dev/null 2>&1
```
This crontab entry executes cgc.sh once every 5 minutes. See the man page for cron to modify the crontab entry to run the CGC at different time intervals.
Save the crontab file.

The cgc.sh script executes a Java class that collects information about each of the JVMs running on the virtual machine and saves it in an area of the virtual machine where the EM4J plug-in for the vSphere Web Client can retrieve it. The vSphere Web Client displays information collected by the most recent CGC execution.

Disabling the vSphere Web Client EM4J Plug-in

You can disable the EM4J plug-in in the vSphere Web Client Plug-in Management console. The plug-in remains installed, but is inoperative. You can re-enable the plug-in later using a similar procedure.

Procedure

From the Applications menu, choose System Administration > Plug-in Management.
Right-click Elastic Memory for Java Client and choose Other > Disable from the context menu.
Click Yes. A Reload vSphere Web Client dialog box appears.
Click Yes.

Viewing Java Workloads in the vSphere Web Client

When the vSphere Web Client EM4J plug-in is installed, select any virtual machine in the inventory tree and then click the Workloads tab.

If the virtual machine has no JVMs, or if EM4J is not properly configured in the virtual machine, the list of workloads is empty and alert messages report why no workloads were discovered.

When JVMs are discovered on the virtual machine, they appear in the workloads list.

The following table describes the data presented in the workloads list.

Table 4. Workloads List

Column	Description
Name	The name of the workload, normally `java`.
Status	The workload status: Normal, Information, or Warning. It is a rollup value, indicating the highest severity of any alerts that exist for the workload.
EM4J Agent Enabled	A checkbox indicating if the EM4J agent is enabled. The EM4J agent is enabled when the ESXi version is supported for EM4J, VMware tools are installed, and the `sched.mem.pshare.guestHintsSyncEnable` configuration parameter is set to `true`.
Main Class	The name of the main Java class the JVM is executing.
Virtual Machine	The name of the virtual machine where the JVM is executing.
Host	The name of the ESXi host executing the virtual machine.

When a JVM is selected in the workloads list, you can select from three tabs to display information about the JVM:

Summary Tab

The Workload Summary tab shows information about the workload selected in the Workloads list.

Status

The Status section shows information about the state of EM4J for workload.

Table 5. Attributes in the Status Section

Attribute	Description
Overall	The overall status for EM4J, either Normal, Information, or Warning.
EM4J Agent Enabled	A checkbox indicating if the EM4J agent is enabled. The EM4J agent is enabled when the ESXi version is supported for EM4J, VMware tools are installed, and the `sched.mem.pshare.guestHintsSyncEnable` configuration parameter is set to `true`.

Annotations

The Annotations section displays descriptive information associated with the workload. If EM4J is able to determine the name of the application the JVM is running, it is displayed here.

JVM Details

The JVM Details section displays information about the Java virtual machine.

Table 6. Attributes in the JVM Details Section

Attribute	Description
PID	The operating system process ID for the executing JVM process.
Main Class	The name of the main Java class the JVM executed on startup.
Large Pages Supported by Guest OS	A checkbox indicating if `/proc/sys/vm/nr_hugepages` exists with non-zero contents, which signals that the guest OS is configured to use large memory pages. Using large memory pages is a best practice for running Java on a virtual machine, but is not a requirement to use EM4J.
Number of Large Pages	The number of large pages configured in the Guest OS; the contents of `/proc/sys/vm/nr_hugepages`.
JVM Using Large Pages	A checkbox indicating if the JVM is using large memory pages.JVM large page support is enabled with the `-XX:+UseLargePages` Java command line option.

Related Items

The Related Items section lists information about the location of the virtual machine in the vSphere Web Client inventory.

Table 7. Attributes in the Related Items Section

Attribute	Description
Virtual Machine	The name of the virtual machine where the workload is executing.
Host	The host name or IP number of the ESXi host.
Resource Pool	The name of the resource pool the virtual machine belongs to.

Alerts Tab

The Alerts tab displays a list of alerts that have been activated in response to the state of a Java workload. Alerts are messages that report real or potential configuration problems that prevent EM4J from operating normally.

EM4J alerts have a severity of Information, Warning, or Java Best Practice. An alert with a severity of Information reports a non-error condition that prevents EM4J from running. For example, if there are no JVMs running on the virtual machine, an alert with Information severity reports the fact.

An alert with Warning severity reports an error condition that prevents EM4J from operating normally. For example, if the Console Guest Collector on a virtual machine is unable to connect to the JMX port of a JVM, an alert with Warning severity is issued.

Java Best Practice alerts are suggestions to help improve performance when running Java on a virtual machine, for example, using large memory pages when the OS supports them.

The Status displayed on the Workloads list is the most severe alert level for the workload, or Normal if there are no alerts. When determining the severity, Java Best Practice alerts are treated as Information alerts.

A list of the potential alerts appears in the Elastic Memory for Java Client Plug-in Online Help topic for the Alerts tab.

Resource Management Tab

The Resource Management tab displays information about memory usage in JVMs and the virtual machine, and JVM garbage collections. The EM4J memory statistics displayed on the Resource Management tab are a snapshot of memory utilization and garbage collection when the Console Guest Collector last ran on the guest OS.

Note

The statistics shown in the Guest Memory section are updated more frequently than the JVM Heap Memory statistics and provide the most up-to-date and accurate measure of ballooned memory size for the virtual machine.

The JVM Heap memory statistics include details reported by all JVMs running when the CGC executed last. When a JVM exits, its balloon memory can persist in the ESXi-managed balloon for the virtual machine, although it is no longer associated with a JVM. For this reason, the total balloon size reported for all EM4J-enabled JVMs may be less than the actual balloon size.

Heap Memory: Selected JVM

This section includes a stacked bar graph depicting the composition of heap memory in the JVM selected in the Workloads list.

Table 8. JVM Heap Memory Statistics

Attribute	Description
Application Heap Memory	Amount of heap memory currently used, excluding EM4J ballooned and shareable memory.
Ballooned Heap Memory (EM4J Balloon)	Amount of the EM4J memory reclaimed by ESXi.
EM4J Shareable Heap Memory	Amount of heap memory objects owned by EM4J that are not currently ballooned, but can still be reclaimed by ESXi as zero-shared memory.
Free Heap Memory	Amount of free heap memory.
Committed Heap Memory	Amount of heap memory currently allocated to the JVM. It varies as the JVM requests additional memory or releases memory to the operating system. Committed Heap Memory is the sum of Application Heap Memory, Ballooned Heap Memory, EM4J Shareable Heap Memory, and Free Heap Memory.

The portion of heap memory that is in use contains objects stored by applications running on the JVM (Application Heap Memory) as well as objects created by EM4J. Of the memory controlled by EM4J, some may be reclaimed by ESXi to satisfy the balloon target for the virtual machine (Ballooned Heap Memory) and the remainder is zero-filled, making it available to ESXi to reclaim with transparent page sharing (EM4J Shareable Memory).

Heap Memory: All EM4J-Enabled JVMs

This section includes a stacked bar graph depicting the composition of heap memory for all EM4J-enabled JVMs running on the virtual machine.

Table 9. JVM Heap Memory Statistics

Attribute	Description
Total Application Heap Memory	Total amount of heap memory used by all JVMs on the virtual machine, excluding EM4J ballooned and shareable memory.
Total Ballooned Heap Memory (EM4J Balloon)	Total amount of heap memory reclaimed by all EM4J balloons on the virtual machine.
Total EM4J Shareable Heap Memory	Total amount of heap memory objects owned by EM4J that are not currently ballooned, but that could be reclaimed by ESXi as zero-shared memory.
Total Free Heap Memory	Total amount of free heap memory for all JVMs on the virtual machine.
Total Committed Heap Memory	Total amount of committed heap memory for all JVMs on the virtual machine.

Guest Memory

The Guest Memory section displays information about guest memory usage. The statistics in this section are live, unlike other sections displayed on this tab, which are from the most recent CGC execution.

Table 10. Guest Memory Statistics

Attribute	Description
Active Guest Memory	Amount of memory recently accessed.
Private Guest Memory	Amount of memory backed by host memory and not being shared.
Shared Guest Memory	Amount of the virtual machine's memory being shared.
Ballooned Guest Memory	Amount of the virtual machine's memory reclaimed by ballooning.
Compressed Guest Memory	Amount of the virtual machine's memory in compression cache.
Swapped Guest Memory	Amount of the virtual machine's memory reclaimed by swapping.
Unaccessed Guest Memory	Amount of the virtual machine's memory never referenced by the guest.

Garbage Collectors

The Garbage Collectors section displays a list of the JVM's garbage collectors.

Table 11. Garbage Collectors Statistics

Attribute	Description
Name	Name of the garbage collector.
Collection Count	Cumulative number of garbage collections.
Collection Time (ms)	Cumulative execution time of the garbage collector, in milliseconds.

Initial Memory Configuration

The Initial Memory Configuration section describes the initial JVM memory layout as configured on the java command line.

Table 12. Initial Memory Configuration Statistics

Attribute	Description
Minimum Heap Size	Minimum heap size, specified with the `-Xms` system property.
Maximum Heap Size	Maximum heap size, specified with the `-Xmx` JVM option.
Maximum PermGen Size	Maximum size of the permanent generation, specified with the `-XX:MaxPermSizeJVM` option.
Thread Stack Size	Size of the stack for each thread, specified with the `-Xss` JVM option.
Using Large Pages	Checked if the JVM is using large memory pages. Specified with the `-XX:+UseLargePages` JVM option. Large memory support must be configured in the OS to enable this option.