tc Server includes a full set of diagnostic features that make it easy to troubleshoot problems with tc Runtime instances and the applications that you deploy to them. For each diagnostic feature, the tc Server Hyperic plug-in has one or more corresponding preconfigured alerts.
After Hyperic triggers an alert associated with a diagnostic feature (because the associated condition has been met), Hyperic disables the alert until an administrator marks it as Fixed. You can use Hyperic to further configure this alert with additional control actions or even disable it, as described in the following sections:
The preconfigured Hyperic alerts associated to the diagnostic features of tc Runtime work on one of two Hyperic resources: either the tc Runtime instance itself, or with a service of the tc Runtime instance. This information is important to know because it determines how you view, and optionally change, a particular alert.
The following table lists each preconfigured alert and the Hyperic
resource type to which it is associated. The resource type
SpringSource tc Runtime 6.0 refers to the tc Runtime
instance itself; the resource type
SpringSource tc Runtime 6.0
, such as
Runtime 6.0 Thread Diagnostics, refers to a service of the tc
Note: The tc Runtime version is associated with the core version of Tomcat on which the runtime is based, rather than the version of the tc Server bundle.
The third column in the table indicates whether the alert is triggered by a metric condition or an event/log level condition. If the former, the name of the metric is displayed; if the latter, the specific string in the log (if any) that triggers the alert is displayed.
Table 2. Preconfigured tc Runtime Alerts
|Alert Name||Associated Hyperic Resource Type||Metric or Events/Log Level Based?|
|Deadlocks Detected||SpringSource tc Runtime 6.0 and SpringSource tc Runtime 7.0||Metric (Deadlocks Detected)|
|Excessive Time Spent in Garbage Collection||SpringSource tc Runtime 6.0 and SpringSource tc Runtime 7.0||Metric (Percent Up Time in Garbage Collection)|
|Slow or Failed Request||SpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Thread Diagnostics||Events/Logs Level.|
|JDBC Connection Abandoned||SpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool Global||Events/Logs Level (CONNECTION ABANDONED)|
|JDBC Connection Failed||SpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool Global||Events/Logs Level (CONNECTION FAILED)|
|JDBC Query Failed||SpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool Global||Events/Logs Level (FAILED QUERY)|
|Slow JDBC Query||SpringSource tc Runtime 6.0 and SpringSource tc Server 7.0 Tomcat JDBC Connection Pool Global||Events/Logs Level (SLOW QUERY)|
The following procedure summarizes how to view and change preconfigured alerts. For a detailed tutorial that shows how to view and change the Deadlocks Detected alert, see "Tutorial: Using Hyperic to Configure and Manage tc Runtime Instances" in Getting Started with vFabric tc Server.
Browse to the resource to which the alert is associated, as described in the preceding table. See "Getting Started with the Hyperic User Interface" in Getting Started with vFabric tc Server for information about browsing to Hyperic resources.
Click the Alert tab.
Click the Configure button. A table of alerts currently configured for the resource is displayed.
Click the name of the alert. The Alert Definition page for the alert is displayed.
The definition page has three sections: the top Alert Properties section provides general properties of the alert; the middle Condition Set section describes the conditions that trigger the alert; and a series of tabs at the bottom enable you to configure the particular control action that occurs if the alert is triggered, the escalation scheme, who should be notified if the alert is triggered, and so on.
If you want to change the general properties, conditions, control actions, and so on of the alert, click the appropriate EDIT... button, make your changes, then click OK.
To disable the alert, go back to the Alert Definitions table,
select the name of the alert by checking the box to the left of its
name, then select
No for the Set
Active drop-down list and click the arrow to the
The remainder of this chapter describes each alert in more detail, including any special instructions to enable the alert.
As shown in the Preconfigured tc Runtime Alerts table, the two alerts associated with the tc Runtime instance itself use metrics in their condition to determine whether the alert should be triggered. The following procedure describes how you can view, and optionally change, the collection interval for Deadlock Detection and Excessive Time in Garbage Collection.
Click the Administration tab at the top of the Hyperic user interface.
In the Hyperic Server Settings section, click the Monitoring Defaults link.
Scroll down until you find the
SpringSource tc Runtime
SpringSource tc Runtime 7.0 entry in the
Server Types table, and then click the EDIT
TEMPLATE METRIC link to the right.
A page shows all metrics associated with the tc Runtime instance. For example, under Utilization you will find the Deadlocks Detected metric. By default, the Collection Interval column shows that Hyperic Server collects information about this metric every 2 minutes.
To change the collection interval for a specific metric, select it by clicking the box to the left of its name.
Enter the new collection interval at the bottom of the page in the Collection Interval for Selected field, specify whether it is in minutes or hours, then click the arrow to the right.
The tc Runtime automatically detects whether a thread deadlock occurs in a tc Runtime instance or an application deployed to the instance.
The out-of-the-box Hyperic alert is triggered if the Deadlocks Detected metric exceeds 0. Hyperic checks the metric every two minutes to see whether the condition is met. Hyperic applies this alert to all auto-discovered tc Runtime instances and enables it by default. This alert is associated with the tc Runtime instance itself.
For a detailed tutorial that shows how to view and change the Deadlocks Detected alert, see "Tutorial: Using Hyperic to Configure and Manage tc Runtime Instances" in Getting Started with vFabric tc Server.
A Hyperic metric represents the percentage of process up time (0 -100) that the tc Runtime instance has spent in garbage collection.
The alert is triggered when the total garbage collection time is excessive (by default, 40% of process up time.) Hyperic checks this metric every 5 minutes to see if the condition has been met. Hyperic applies this alert to all auto-discovered tc Runtime instances and enables it by default.
When clients begin connecting and using a Web application deployed to a tc Runtime instance, they may encounter slow or failed requests. Although the tc Runtime instance logs these errors in the log files by default, it is often difficult to pinpoint the exact origin of the error and how to go about fixing it. By enabling thread diagnostics, tc Runtime provides additional information to help you troubleshoot the problem.
A failed request is one that simply did not execute; a slow request is a request that takes longer than a certain threshold. The default threshold is 500 milliseconds.
When you enable thread diagnostics, you can view the following contextual information about a slow or failed client request:
Time and date of the slow or failed request.
Exact URL invoked by the client that resulted in a slow or failed request.
Exact error returned by the request.
Database queries that were executed as part of the request and how long each one took.
Whether any database connection failed or succeeded.
Whether the database had any other connectivity problems.
Whether the database connection pool ran out of connections.
Whether any garbage collection occurred during the request, and if so, how long it took.
The associated Hyperic alert is triggered if a client request to tc Runtime is slow (over a configured threshold) or if it failed.
This alert is not enabled by default. Explicitly enable it as follows:
Browse to the Views > Server Configuration console page for the tc Runtime instance.
Click the Services tab.
In the table, click the service you want to configure; the
default tc Runtime service is called
In the Thread Diagnostics section, check the Enable Thread Diagnostics property.
At the bottom of the page, click Save.
Click the necessary links and buttons to push configuration changes to the tc Runtime instance and restart the instance.
Hyperic includes a new service called
Runtime 6.0 Tomcat JDBC Connection Pool Global that represents
any high-concurrency Tomcat JDBC datasources you might have configured
for your tc Runtime instance. This service monitors the health of the
datasource, such as whether its connection to the database has failed or
was abandoned, and whether the JDBC queries that clients execute are
taking too long. Hyperic creates this service when you create a new
Tomcat JDBC datasource; one instance of a service exists per
Four Hyperic alerts are associated with this diagnostic feature; they are triggered as follows:
JDBC Connection Failed: A particular high-concurrency JDBC connection that uses a configured datasource fails.
JDBC Connection Abandoned: A particular high-concurrency JDBC connection that uses a configured datasource is abandoned by the database server.
JDBC Query Failed: A high-concurrency JDBC query fails.
Slow JDBC Query: A high-concurrency JDBC query takes too long to execute.
To receive monitoring information for the preceding JDBC alerts, enable log tracking for this service:
Browse to the
SpringSource tc Runtime 6.0 Tomcat JDBC
Connection Pool Global service associated with your JDBC
Click the Inventory tab.
In the Configuration Properties section, be sure that the service.log_track.enable property is checked. Checking this box subscribes Hyperic to JMX notifications sent from the tc Runtime instance, which then get displayed in Hyperic as log events.