[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster] client stops receiving messages



Hi Marshal,
could you send to me (laghi at swissinfo.org) and marcel (mr at marcelruff.info) the complete logs (if you have it possibily with traces) of xmlBlaster ?


Also the publishQos of the messages would be interesting. (to get them you could temporarly switch dump[socket] on).

Regards
Michele



Marshall Shapiro wrote:
Marcel,

The session is not persistent. Neither are the subscriptions. (I believe)
the messages are also not persistent. Most have a very short lifetime. (The
idea is to deliver to those listening and get rid of it). (I didn't
implement all the clients so I can't be 100% sure that no client is
mistakenly publishing persistent messages, but I don't think so).

I'm not using a database, just flat files for storage. Is that a problem?
(Btw, am I losing performance because of that? If I switch to a DB, do you
recommend postgres or mysql?)

The setting in xmlBlaster.properties related to cache size are:

--------------------
# How many history messages of same key oid (topic) to hold in xmlBlaster
(holds references only)
queue/history/maxEntriesCache = 10
queue/history/maxEntries = ${queue/history/maxEntriesCache}

# How big is a callback queue for each client (holds references only)
queue/callback/maxEntriesCache = 1000
queue/callback/maxEntries = ${queue/callback/maxEntriesCache}

# How big is the topic cache for messages of same oid (here your message
content occupies the memory)
persistence/msgUnitStore/maxEntriesCache = 10000
persistence/msgUnitStore/maxEntries = 100000000
persistence/msgUnitStore/maxBytesCache = 2097152
persistence/msgUnitStore/maxBytes = 2147483647

# Tail back queue on client side (contains the real data of all message
types)
queue/connection/maxEntriesCache = 1000
queue/connection/maxEntries = 10000000
--------------------

The callback queue is not full when the client stops getting messages, but
it then fills up quickly. (Originally I thought the problem was that the
callback queue was full, but after watching more carefully I saw that the
client stops getting the messages first, and then, because of that, the
queue fills up). I'm not sure what you mean by the callback queue swapping.


Thanks again. Let me know if still need more information. Any insight into what is going on is appreciated.


Marshall


-----Original Message----- From: owner-xmlblaster at server.xmlblaster.org [mailto:owner-xmlblaster at server.xmlblaster.org] On Behalf Of Marcel Ruff Sent: Tuesday, October 11, 2005 7:13 AM To: xmlblaster at server.xmlblaster.org Subject: Re: [xmlblaster] client stops receiving messages

Hi,

which persistence configuration do you have, are the message persistent?
Which database and pool sizes (like queue/callback/maxEntriesCache etc from xmlBlaster.properties).


Is the callback queue filled and swapping (checking with JMX)?

thanks
Marcel

Marshall Shapiro wrote:


I am working on a project that is using xmlBlaster.
I have one client that stops receiving messages in the face of very high message rates and size. This client is observing all the messages in the system, so it is receiving both lot's of small messages (~2/sec) and some number of large messages (2-10 MB). At some peak moment in a run the client will stop receiving messages and doesn't start receiving them again. Using the jmx web page I can see that the session is still there and subscribed.


The logs I'm going to show you are from a test I ran in which I was trying to break it. I ran the system normally for a while and then threw a bunch of large files at it (~9 MB each). I got the following errors. These are typical of the errors I see in the logs when the client breaks.

I was hoping if I showed you excerpts from the logs, you could help me figure out what's going on with this client.


First I got some messages like this:

[Oct 10, 2005 4:23:42 PM ERROR XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0


MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/
-4/NORM/1128975795940000000/ComponentHeartbeat]


No no meat found, incr=-1



Then I got a bunch of client side exceptions such as:

[Oct 10, 2005 4:26:36 PM WARN  XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 RequestBroker] Generating dead message


'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289759916
09000002/ComponentHeartbeat'


from publisher=/node/xmlBlaster_195_1_1_41_3412/client/PD/-9 because delivery with queue 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4' failed: XmlBlasterException errorCode=[user.update.internalError] serverSideException=false location=[AbstractCallbackExtended.update] message=[#exported Error in client code : java.lang.NullPointerException] [See URL


_http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.list ing.html#user.update.internalError_]



Then it got lots more of these errors. (These messages actually always show up with the problem):

[Oct 10, 2005 4:30:08 PM ERROR XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41 3412/client/jaguar/-4/NORM/1128976172075000002/ComponentHeartbeat] No no meat found, incr=-1



Followed by lots of these (I've often seen these null queue messages with this problem):

[Oct 10, 2005 4:44:45 PM WARN  XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-1 RequestBroker] Generating dead message


'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289770086
60000000/SystemHeartbeat'


from publisher=/node/xmlBlaster_195_1_1_41_3412/client/MC/-6 because delivery with queue 'null' failed: XmlBlasterException serverSideException=true node=[xmlBlaster_195_1_1_41_3412] location=[MsgQueueUpdateEntry]

stackTrace=errorCode=internal.unknown message=#exported Message 1128977060215000002 not found
at



org.xmlBlaster.engine.queuemsg.ReferenceEntry.getMsgUnit(ReferenceEntry.java
:289)


at


org.xmlBlaster.engine.dispatch.CbDispatchConnection.doSend(CbDispatchConnect
ion.java:135)


at


org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.java
:221)


at


org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchConnect
ionsHandler.java:420)


at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:166)
at



EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.ja
va:732)


       at java.lang.Thread.run(Thread.java:536)


versionInfo=version=1.0.6,revision=exported,os.name=Linux,os.version=2.4.21-
15.EL,java.vm.vendor=Blackdown


Java-Linux


Team,java.vm.version=Blackdown-1.4.1-01,os.arch=i386,build.timestamp=08/10/2
005


01:45 PM,build.java.vendor=Sun Microsystems Inc.,build.java.version=1.4.2_06

errorCode


description=http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.erro rcodes.listing.html#internal.unknown



I'm running xmlBlaster version 1.0.6. The client is written in Java, compiled using Java 1.4.2.

Let me know if you need more information.

Thank you in advance for your help.

Marshall