Re: [xmlblaster] Callback message queue fills up

[Marcel Ruff <mr at marcelruff.info>]

Hi David,

do you have a jconsole to observe the two nodes?

If yes, please check the number of subscriptions the node A has 
forwarded to node B
(look into node B and check the number of subscriptions of client A) 
during such a case.
In case the subscribeQos has set

<multiSubscribe>true</multiSubscribe>

(which is the default) it could be that the subscriptions multiplied
during small connection errors and reconnects.
This is just a guess.
If it is the case please set multiSubscribe to false.

Is there a high CPU load during the 1001 message case?
Are the hearbeat messages persistent messages?
Was the client connected or offline during this message overflow?
Does your heartbeat have a unique id so that you can tell for sure if the same
published message is cloned many times (try a peek on the callback queue with jconsole)?

A final option is to use the current svn xmlBlaster and switch on the checkpoint logging
to get a better idea what is going on.

And finally it could be a problem with your client not taking the callback messages.

Another idea: The callback queue contains only a reference on the message.
If it expires the message-'meat' is destroyed but the reference remains in the queue
until it is looked at during delivery (and then thrown to garbage), Michele, could this be?

thanks
Marcel


David R Robison wrote:
> We are experiencing something strange in xmlBlaster 1.6.1. We have two 
> nodes, node A subscribes to messages from node B. These are heartbeat 
> messages and are generated every 15 seconds with a lifetime of 30 
> seconds. A client connects to node A and subscribes to the messages, 
> node A then passes the subscription onto node B. Watching the callback 
> message queue, everything seems to run well, at most 1 message in the 
> queue waiting to be sent. It can run like this for days. Then, 
> unexpectedly, the callback queue will show as being full (in this case 
> 1001 messages). The queue contains many duplicated messages with 
> different timestamps. From there, the server struggles to deliver the 
> messages and keep the queue empty. The reader never seems to read 
> enough messages to get the queue back down to zero. If I stop the 
> client and reconnect, it will recreate its queue and be back to 
> normal. I know this is a bit sketchy, but it is becoming a real 
> problem for us.
>
> Any thoughts on what might be the problem? Any idea of where to start 
> looking?
>
> One more note, when the client is subscribing to heartbeats that are 
> generated on Node A, the client never fails in this manor, only when 
> it is subscribing to node A for a message generated on node B.
>
> Thanks, in advance,
> David Robison

--
Marcel Ruff
http://www.xmlBlaster.org