[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [xmlblaster] client stops receiving messages

To: xmlblaster at server.xmlBlaster.org
Subject: RE: [xmlblaster] client stops receiving messages
From: Marshall Shapiro <mshapiro at draper.com>
Date: Tue, 11 Oct 2005 11:18:15 -0400
In-reply-to: <434B9E4E.2080802 at marcelruff.info>
Organization: Draper Laboratory
Reply-to: xmlblaster at server.xmlBlaster.org
Sender: owner-xmlblaster at server.xmlBlaster.org
Thread-index: AcXOVnlye6DdkFLrRZmGR7rbdh+TWgAHau/A

Marcel,

The session is not persistent. Neither are the subscriptions. (I believe)
the messages are also not persistent. Most have a very short lifetime. (The
idea is to deliver to those listening and get rid of it). (I didn't
implement all the clients so I can't be 100% sure that no client is
mistakenly publishing persistent messages, but I don't think so).

I'm not using a database, just flat files for storage. Is that a problem?
(Btw, am I losing performance because of that? If I switch to a DB, do you
recommend postgres or mysql?)

The setting in xmlBlaster.properties related to cache size are:

--------------------
# How many history messages of same key oid (topic) to hold in xmlBlaster
(holds references only)
queue/history/maxEntriesCache = 10
queue/history/maxEntries = ${queue/history/maxEntriesCache}

# How big is a callback queue for each client (holds references only)
queue/callback/maxEntriesCache = 1000
queue/callback/maxEntries = ${queue/callback/maxEntriesCache}

# How big is the topic cache for messages of same oid (here your message
content occupies the memory)
persistence/msgUnitStore/maxEntriesCache = 10000
persistence/msgUnitStore/maxEntries = 100000000
persistence/msgUnitStore/maxBytesCache = 2097152
persistence/msgUnitStore/maxBytes = 2147483647

# Tail back queue on client side (contains the real data of all message
types)
queue/connection/maxEntriesCache = 1000
queue/connection/maxEntries = 10000000
--------------------

The callback queue is not full when the client stops getting messages, but
it then fills up quickly. (Originally I thought the problem was that the
callback queue was full, but after watching more carefully I saw that the
client stops getting the messages first, and then, because of that, the
queue fills up). I'm not sure what you mean by the callback queue swapping.


Thanks again. Let me know if still need more information. Any insight into
what is going on is appreciated.


Marshall


-----Original Message-----
From: owner-xmlblaster at server.xmlblaster.org
[mailto:owner-xmlblaster at server.xmlblaster.org] On Behalf Of Marcel Ruff
Sent: Tuesday, October 11, 2005 7:13 AM
To: xmlblaster at server.xmlblaster.org
Subject: Re: [xmlblaster] client stops receiving messages

Hi,

which persistence configuration do you have, are the message persistent?
Which database and pool sizes (like queue/callback/maxEntriesCache etc 
from xmlBlaster.properties).

Is the callback queue filled and swapping (checking with JMX)?

thanks
Marcel

Marshall Shapiro wrote:

> I am working on a project that is using xmlBlaster.
> I have one client that stops receiving messages in the face of very 
> high message rates and size. This client is observing all the messages 
> in the system, so it is receiving both lot's of small messages 
> (~2/sec) and some number of large messages (2-10 MB). At some peak 
> moment in a run the client will stop receiving messages and doesn't 
> start receiving them again. Using the jmx web page I can see that the 
> session is still there and subscribed.
>
> The logs I'm going to show you are from a test I ran in which I was 
> trying to break it. I ran the system normally for a while and then 
> threw a bunch of large files at it (~9 MB each). I got the following 
> errors. These are typical of the errors I see in the logs when the 
> client breaks.
>
> I was hoping if I showed you excerpts from the logs, you could help me 
> figure out what's going on with this client.
>
>
> First I got some messages like this:
>
> [Oct 10, 2005 4:23:42 PM [31;40mERROR[0m 
> XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
>
MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/
-4/NORM/1128975795940000000/ComponentHeartbeat] 
> No no meat found, incr=-1
>
>
>
> Then I got a bunch of client side exceptions such as:
>
> [Oct 10, 2005 4:26:36 PM [33;40mWARN [0m 
> XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
> RequestBroker] Generating dead message 
>
'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289759916
09000002/ComponentHeartbeat' 
> from publisher=/node/xmlBlaster_195_1_1_41_3412/client/PD/-9 because 
> delivery with queue 
> 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4' failed: 
> XmlBlasterException errorCode=[user.update.internalError] 
> serverSideException=false location=[AbstractCallbackExtended.update] 
> message=[#exported Error in client code : 
> java.lang.NullPointerException] [See URL 
>
_http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.list
ing.html#user.update.internalError_]
>
>
>
> Then it got lots more of these errors. (These messages actually always 
> show up with the problem):
>
> [Oct 10, 2005 4:30:08 PM [31;40mERROR[0m 
> XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
> MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41 
> 3412/client/jaguar/-4/NORM/1128976172075000002/ComponentHeartbeat] No 
> no meat found, incr=-1
>
>
>
> Followed by lots of these (I've often seen these null queue messages 
> with this problem):
>
> [Oct 10, 2005 4:44:45 PM [33;40mWARN [0m 
> XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-1 
> RequestBroker] Generating dead message 
>
'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289770086
60000000/SystemHeartbeat' 
> from publisher=/node/xmlBlaster_195_1_1_41_3412/client/MC/-6 because 
> delivery with queue 'null' failed: XmlBlasterException 
> serverSideException=true node=[xmlBlaster_195_1_1_41_3412] 
> location=[MsgQueueUpdateEntry]
>
> stackTrace=errorCode=internal.unknown message=#exported Message 
> 1128977060215000002 not found
>         at 
>
org.xmlBlaster.engine.queuemsg.ReferenceEntry.getMsgUnit(ReferenceEntry.java
:289) 
>
>         at 
>
org.xmlBlaster.engine.dispatch.CbDispatchConnection.doSend(CbDispatchConnect
ion.java:135) 
>
>         at 
>
org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.java
:221) 
>
>         at 
>
org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchConnect
ionsHandler.java:420) 
>
>         at 
> org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:166)
>         at 
>
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.ja
va:732) 
>
>         at java.lang.Thread.run(Thread.java:536)
>
versionInfo=version=1.0.6,revision=exported,os.name=Linux,os.version=2.4.21-
15.EL,java.vm.vendor=Blackdown 
> Java-Linux 
>
Team,java.vm.version=Blackdown-1.4.1-01,os.arch=i386,build.timestamp=08/10/2
005 
> 01:45 PM,build.java.vendor=Sun Microsystems 
> Inc.,build.java.version=1.4.2_06
>
> errorCode 
>
description=http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.erro
rcodes.listing.html#internal.unknown
>
>
>
> I'm running xmlBlaster version 1.0.6. The client is written in Java, 
> compiled using Java 1.4.2.
>
> Let me know if you need more information.
>
> Thank you in advance for your help.
>
> Marshall
>

Follow-Ups:
- Re: [xmlblaster] client stops receiving messages
  - From: Michele <laghi at swissinfo.org>
- Re: [xmlblaster] client stops receiving messages
  - From: Marcel Ruff <mr at marcelruff.info>

References:
- Re: [xmlblaster] client stops receiving messages
  - From: Marcel Ruff <mr at marcelruff.info>

Prev by Date: Re: [xmlblaster] Change xmlBlaster to JDK 1.5 (and above) only
Next by Date: Re: [xmlblaster] client stops receiving messages
Previous by thread: Re: [xmlblaster] client stops receiving messages
Next by thread: Re: [xmlblaster] client stops receiving messages
Index(es):
- Date
- Thread