[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [xmlblaster] WG: Help! OutOfMemory by publishing lot big topics
<homage>
I must say, as I ran across xmlblaster I found it more and more
interesting as I tried to dig deeper to find out it's intention and
usage. What I liked was the implemented development process, that
surfaced through the reference documentation. I liked the mailing list
quality, it assured me, that this product is well alive and has a good
foundation.
I was also astonished of the number of the clients and protocols
implemented.
I would have been happy to find a short introduction to the concepts
that base the architecture of xmlBlaster, as the reference is sorted by
the requirement 'package identifier'. But I also knew, the lack of a
tutorial is a guarantee to find a community of smart and skilled
developers and users not needing such a introduction to hard ;-) So the
mailing list archive was the way to start with for the glue between the
reference pages.
</homage>
I am happy not to patch anything in the swapping code, as it would be a
real behaviour change that I expect to have unwanted side effects. I
have expected to get some more understanding of how the caching works
and find where to block the message data caching.
I would be happy if I could get some hints for a few API usage questions
I have:
1) Lot of topics:
-----------------
I have chosen to use many topics, as:
a) I (partly) need to process the messages in random order
and
b) I have not figured out how to "get" the topic history, seems I get
only the last message in the topic. I could use the iteration of the
topic history at least where I introduced the splitting of the content
in multiple publishes. There I have used the faster, because known, way
of different topics.
2) StoragePlugin vs. QueuePlugin
------------------------------------------
I have expected to get rid of the 'meat' caching by configuring the
StoragePlugin accordingly before. I just got stock before with the
max*Cache settings. Now, from the following to configurations I expect
the first to swap the message data/meat directly to the persistent
store:
StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
QueuePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
For my understanding: Is the second line suppressing the history,
callback, subject and (if client) connection queue RAM caching of only
the references into the StoragePlugin? Or is it necessary for stopping
the 'meat' caching, too?
Regards,
Johann Thomas
-----Original Message-----
Von: owner-xmlblaster at server.xmlBlaster.org
[mailto:owner-xmlblaster at server.xmlBlaster.org] Im Auftrag von Marcel
Ruff
Gesendet: Montag, 19. Juni 2006 15:49
An: xmlblaster at server.xmlBlaster.org
Betreff: Re: [xmlblaster] WG: Help! OutOfMemory by publishing lot big
topics
Thomas, Johann wrote:
> ...
> - the content dumped is cached "somewhere" in the heap, but it can not
> be garbage collected (each message has it' unique oid).
>
This is the problem: The topics have no swapping mechanism so using a
lot of different topics wont scale. Usually you should choose some well
defined topics (typically 5-100, not more than say 1000).
Each topic can hold any numbers of message instances, they will be
swapped away, depending on the configuration.
As you mentioned above - swapping does hard disk IO and is therefore not
as performing as memory based messages - but it will prevent out of
memory problems.
> - I have set some properties maxEntriesCache = 0, but it had no effect
> at publishing time. I have found some comments in the some of the
> code, saying first overload of queue is allowed.
>
You can just switch off the CACHE and work directly with the persistent
queue:
StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
QueuePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
Now no RAM queue is used as all, the 'CACHE' instantiates the JDBC queue
implementation ...
so no patching is needed.
> - I have patched CacheQueueInterceptorPlugin for me, so that it does
not
> overload the cache above the specified settings. I'm not sure the
server
> is behaving as stable as before with this settings, btw.
> - after the restart of the xmlBlaster server the heap was mostly free.
>
> So I suppose, there some other queues/caches were the first published
> message of a given oid is held in memory, but not preloaded at a
server
> restart.
>
We have one storage holding the message 'meat' (which is reference
counted):
StoragePlugin
and several queues, which have a reference on the 'meat' only:
history
callback
subject
connection (on client side only)
The "callback" queue exists once for each login session, it is the most
important.
The "history" queue is sort of a memory of already processed messages,
and the
"subject" queue is for the seldom use case of PtP messages to a login
name destination.
regards
Marcel
>
> Here the overview of my little patch (between 2006-06-16 2101 JTh
patch
> comments):
>
> // .................... (cut for keeping the email short) package
> org.xmlBlaster.engine.msgstore.cache;
> public class PersistenceCachePlugin implements I_StoragePlugin,
> I_StorageProblemListener, I_Map, PersistenceCachePluginMBean {
>
> // .................... (cut for keeping the email short)
>
> private void assureTransientSpace(I_MapEntry mapEntry) throws
> XmlBlasterException {
>
> while (!spaceLeft(mapEntry, this.transientStore)) {
>
> // .................... (cut for keeping the email short)
>
> if (oldest == null) {
> if (log.isLoggable(Level.FINE)) log.fine("The RAM queue is
> full, new entry '" + mapEntry.getUniqueId() + "' seems to be the first
> and only one, so we accept it");
> break;
> }
> // ++++ 2006-06-16 2101 JTh patch start
> if (oldest == mapEntry) {
> if (log.isLoggable(Level.FINE)) log.fine("2006-06-16 2101
> JTh patch: The RAM queue will become full, new entry '" +
> mapEntry.getUniqueId() + "' seems to be the first and only one, we
still
> removed it, transientStore.numOfEntries now " +
> this.transientStore.getNumOfEntries());
> }
> // ---- 2006-06-16 2101 JTh patch end
> if (log.isLoggable(Level.FINER)) log.finer("Swapping '" +
> oldest.getLogId() + "' to HD ...");
>
> // .................... (cut for keeping the email short)
>
> }
> }
>
> public int put(I_MapEntry mapEntry) throws XmlBlasterException {
>
> // .................... (cut for keeping the email short)
>
> synchronized(this) {
>
> // .................... (cut for keeping the email short)
>
> // separate persistent from transient entries and store the
> persistents in persistence // .................... (cut for keeping
the
> email short)
>
>
> assureTransientSpace(mapEntry);
>
> numTransientPut = this.transientStore.put(mapEntry);
>
> // ++++ 2006-06-16 2101 JTh patch start
> assureTransientSpace(mapEntry);
>
> // ---- 2006-06-16 2101 JTh patch end
> } // sync(this)
> // .................... (cut for keeping the email short)
> }
> }
>
> Regards,
>
> Johann Thomas
>
> --
> OpenPGP: 0x224225EA (7454 D7BB A75D EA8E 5DFF EA4E 778E A85B 2242
25EA)
>
>
>