Managing tc Runtime-Related Hyperic Alerts

tc Server includes a full set of diagnostic features that make it easy to troubleshoot problems with tc Runtime instances and the applications that you deploy to them. For each diagnostic feature, the tc Server Hyperic plug-in has one or more corresponding preconfigured alerts.

After Hyperic triggers an alert associated with a diagnostic feature (because the associated condition has been met), Hyperic disables the alert until an administrator marks it as Fixed. You can use Hyperic to further configure this alert with additional control actions or even disable it, as described in the following sections:

Viewing and Changing the Preconfigured Alerts

The preconfigured Hyperic alerts associated to the diagnostic features of tc Runtime work on one of two Hyperic resources: either the tc Runtime instance itself, or with a service of the tc Runtime instance. This information is important to know because it determines how you view, and optionally change, a particular alert.

The following table lists each preconfigured alert and the Hyperic resource type to which it is associated. The resource type SpringSource tc Runtime 6.0 refers to the tc Runtime instance itself; the resource type SpringSource tc Runtime 6.0 Service, such as SpringSource tc Runtime 6.0 Thread Diagnostics, refers to a service of the tc Runtime instance.

Note: The tc Runtime version is associated with the core version of Tomcat on which the runtime is based, rather than the version of the tc Server bundle.

The third column in the table indicates whether the alert is triggered by a metric condition or an event/log level condition. If the former, the name of the metric is displayed; if the latter, the specific string in the log (if any) that triggers the alert is displayed.

Table 2. Preconfigured tc Runtime Alerts

Alert NameAssociated Hyperic Resource TypeMetric or Events/Log Level Based?
Deadlocks DetectedSpringSource tc Runtime 6.0 and SpringSource tc Runtime 7.0Metric (Deadlocks Detected)
Excessive Time Spent in Garbage CollectionSpringSource tc Runtime 6.0 and SpringSource tc Runtime 7.0Metric (Percent Up Time in Garbage Collection)
Slow or Failed RequestSpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Thread DiagnosticsEvents/Logs Level.
JDBC Connection AbandonedSpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool GlobalEvents/Logs Level (CONNECTION ABANDONED)
JDBC Connection FailedSpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool GlobalEvents/Logs Level (CONNECTION FAILED)
JDBC Query FailedSpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool GlobalEvents/Logs Level (FAILED QUERY)
Slow JDBC QuerySpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool GlobalEvents/Logs Level (SLOW QUERY)

The following procedure summarizes how to view and change preconfigured alerts. For a detailed tutorial that shows how to view and change the Deadlocks Detected alert, see "Tutorial: Using Hyperic to Configure and Manage tc Runtime Instances" in Getting Started with vFabric tc Server.

  1. Browse to the resource to which the alert is associated, as described in the preceding table. See "Getting Started with the Hyperic User Interface" in Getting Started with vFabric tc Server for information about browsing to Hyperic resources.

  2. Click the Alert tab.

  3. Click the Configure button. A table of alerts currently configured for the resource is displayed.

  4. Click the name of the alert. The Alert Definition page for the alert is displayed.

    The definition page has three sections: the top Alert Properties section provides general properties of the alert; the middle Condition Set section describes the conditions that trigger the alert; and a series of tabs at the bottom enable you to configure the particular control action that occurs if the alert is triggered, the escalation scheme, who should be notified if the alert is triggered, and so on.

  5. If you want to change the general properties, conditions, control actions, and so on of the alert, click the appropriate EDIT... button, make your changes, then click OK.

  6. To disable the alert, go back to the Alert Definitions table, select the name of the alert by checking the box to the left of its name, then select No for the Set Active drop-down list and click the arrow to the right.

The remainder of this chapter describes each alert in more detail, including any special instructions to enable the alert.

Viewing and Changing the Metric Collection Interval

As shown in the Preconfigured tc Runtime Alerts table, the two alerts associated with the tc Runtime instance itself use metrics in their condition to determine whether the alert should be triggered. The following procedure describes how you can view, and optionally change, the collection interval for Deadlock Detection and Excessive Time in Garbage Collection.

  1. Click the Administration tab at the top of the Hyperic user interface.

  2. In the Hyperic Server Settings section, click the Monitoring Defaults link.

  3. Scroll down until you find the SpringSource tc Runtime 6.0 or SpringSource tc Runtime 7.0 entry in the Server Types table, and then click the EDIT TEMPLATE METRIC link to the right.

    A page shows all metrics associated with the tc Runtime instance. For example, under Utilization you will find the Deadlocks Detected metric. By default, the Collection Interval column shows that Hyperic Server collects information about this metric every 2 minutes.

  4. To change the collection interval for a specific metric, select it by clicking the box to the left of its name.

  5. Enter the new collection interval at the bottom of the page in the Collection Interval for Selected field, specify whether it is in minutes or hours, then click the arrow to the right.

Deadlock Detection

The tc Runtime automatically detects whether a thread deadlock occurs in a tc Runtime instance or an application deployed to the instance.

The out-of-the-box Hyperic alert is triggered if the Deadlocks Detected metric exceeds 0. Hyperic checks the metric every two minutes to see whether the condition is met. Hyperic applies this alert to all auto-discovered tc Runtime instances and enables it by default. This alert is associated with the tc Runtime instance itself.

For a detailed tutorial that shows how to view and change the Deadlocks Detected alert, see "Tutorial: Using Hyperic to Configure and Manage tc Runtime Instances" in Getting Started with vFabric tc Server.

Excessive Time in Garbage Collection

A Hyperic metric represents the percentage of process up time (0 -100) that the tc Runtime instance has spent in garbage collection.

The alert is triggered when the total garbage collection time is excessive (by default, 40% of process up time.) Hyperic checks this metric every 5 minutes to see if the condition has been met. Hyperic applies this alert to all auto-discovered tc Runtime instances and enables it by default.

Enabling the Slow or Failed Request Alert

When clients begin connecting and using a Web application deployed to a tc Runtime instance, they may encounter slow or failed requests. Although the tc Runtime instance logs these errors in the log files by default, it is often difficult to pinpoint the exact origin of the error and how to go about fixing it. By enabling thread diagnostics, tc Runtime provides additional information to help you troubleshoot the problem.

A failed request is one that simply did not execute; a slow request is a request that takes longer than a certain threshold. The default threshold is 500 milliseconds.

When you enable thread diagnostics, you can view the following contextual information about a slow or failed client request:

  • Time and date of the slow or failed request.

  • Exact URL invoked by the client that resulted in a slow or failed request.

  • Exact error returned by the request.

  • Database queries that were executed as part of the request and how long each one took.

  • Whether any database connection failed or succeeded.

  • Whether the database had any other connectivity problems.

  • Whether the database connection pool ran out of connections.

  • Whether any garbage collection occurred during the request, and if so, how long it took.

The associated Hyperic alert is triggered if a client request to tc Runtime is slow (over a configured threshold) or if it failed.

This alert is not enabled by default. Explicitly enable it as follows:

  1. Browse to the Views > Server Configuration console page for the tc Runtime instance.

  2. Click the Services tab.

  3. In the table, click the service you want to configure; the default tc Runtime service is called Catalina.

  4. In the Thread Diagnostics section, check the Enable Thread Diagnostics property.

  5. At the bottom of the page, click Save.

  6. Click the necessary links and buttons to push configuration changes to the tc Runtime instance and restart the instance.

Enabling JDBC Connection Monitoring

Hyperic includes a new service called SpringSource tc Runtime 6.0 Tomcat JDBC Connection Pool Global that represents any high-concurrency Tomcat JDBC datasources you might have configured for your tc Runtime instance. This service monitors the health of the datasource, such as whether its connection to the database has failed or was abandoned, and whether the JDBC queries that clients execute are taking too long. Hyperic creates this service when you create a new Tomcat JDBC datasource; one instance of a service exists per datasource.

Four Hyperic alerts are associated with this diagnostic feature; they are triggered as follows:

  • JDBC Connection Failed: A particular high-concurrency JDBC connection that uses a configured datasource fails.

  • JDBC Connection Abandoned: A particular high-concurrency JDBC connection that uses a configured datasource is abandoned by the database server.

  • JDBC Query Failed: A high-concurrency JDBC query fails.

  • Slow JDBC Query: A high-concurrency JDBC query takes too long to execute.

To receive monitoring information for the preceding JDBC alerts, enable log tracking for this service:

  1. Browse to the SpringSource tc Runtime 6.0 Tomcat JDBC Connection Pool Global service associated with your JDBC datasource.

  2. Click the Inventory tab.

  3. In the Configuration Properties section, be sure that the service.log_track.enable property is checked. Checking this box subscribes Hyperic to JMX notifications sent from the tc Runtime instance, which then get displayed in Hyperic as log events.