Durable Client/Server Event Messaging

Initial Operation

The initial startup of a durable client is similar to the startup of any other client, except that it specifically calls the ClientCache.readyForEvents method when all regions, listeners, and CQs on the client are ready to process messages from the server.

Disconnection

While the client and servers are disconnected, their operation varies depending on the circumstances.

Normal disconnect. When a client closes its connection, the servers stop sending messages to the client and release its connection. If the client requests it, the servers maintain the queues and durable interest list and CQ information until the client reconnects or times out. The non-durable interest lists and CQs are discarded. The servers continue to queue up incoming messages for entries on the durable interest list and in the durable CQ list. All messages that were in the queue when the client disconnected remain in the queue. If the client requests not to have its subscriptions maintained, or if there are no durable subscriptions or CQs, the servers unregister the client and do the same cleanup as for a non-durable client.
Abnormal disconnect. If the client crashes or loses its connections to all servers, the servers automatically maintain its message queue and durable subscriptions and CQs until it reconnects or times out.
Client disconnected but operational. If the client operates while it is disconnected, it gets what data it can from the local client cache. Since updates are not allowed, the data can become stale. An UnconnectedException occurs if an update is attempted.
Client stays disconnected past timeout period. The servers track how long to keep a durable subscription queue alive based on the durable-client-timeout setting. If the client remains disconnected longer than the timeout, the servers unregister the client and do the same cleanup that is performed for a non-durable client. The servers also log an alert. When a timed-out client reconnects, the servers treat it as a new client making its initial connection.

Reconnection

During initialization, the client cache is not blocked from doing operations, so you might be receiving old stored events from the server at the same time that your client cache is being updated by much more current events. These are the things that can act on the cache concurrently:

Results returned by the server in response to the client’s interest registrations and CQ executions.
Client cache operations by the application.
Callbacks triggered by replaying old events from the queue

GemFire handles the conflicts between the application and interest registrations and CQ executions do not create cache update conflicts. But you must program your event handlers so they don't conflict with current operations. This is true for all event handlers, but it is especially important for those used in durable clients. Your handlers may receive events well after the fact and you must ensure your programming takes that into account.

This figure shows the three concurrent procedures during the initialization process. The application begins operations immediately on the client (step 1), while the client’s cache ready message (also step 1) triggers a series of queue operations on the servers (starting with step 2 on the primary server). At the same time, the client registers interest (step 2 on the client) and receives a response from the server. Message B2 applies to an entry in Region A, so the cache listener handles B2’s event. Because B2 comes before the marker, the client does not apply the update to the cache.

Durable client reconnection.

Durable Event Replay

When a durable client reconnects before the timeout period, the servers replay the events that were stored while the client was gone and then resume normal event messaging to the client. To avoid overwriting current entries with old data, the stored events are not applied to the client cache. Stored events are distinguished from new normal events by a marker that is sent to the client once all old events are replayed.

All servers with a queue for this client place a marker in their queue when the client reconnects.
The primary server sends the queued messages to the client up to the marker.
The client receives the messages but does not apply the usual automatic updates to its cache. If cache listeners are installed, they handle the events.
The client receives the marker message indicating that all past events have been played back.
The server sends the current list of live regions.
For every CacheListener in each live region on the client, the marker event triggers the afterRegionLive callback. After the callback, the client begins normal processing of events from the server and applies the updates to its cache.

Even when a new client starts up for the first time, the client cache ready markers are inserted in the queues. If messages start coming into the new queues before the servers insert the marker, those messages are considered as having happened while the client was disconnected, and their events are replayed the same as in the reconnect case.

Application Operations During Interest Registration

Application operations take precedence over interest registration responses. The client can perform operations while it is receiving its interest registration responses. When adding register interest responses to the client cache, the following rules are applied:

If the entry already exists in the cache with a valid value, it is not updated.
If the entry is invalid, and the register interest response is valid, the valid value is put into the cache.
If an entry is marked destroyed, it is not updated. Destroyed entries are removed from the system after the register interest response is completed.
If the interest response does not contain any results, because all of those keys are absent from the server’s cache, the client’s cache can start out empty. If the queue contains old messages related to those keys, the events are still replayed in the client’s cache.