[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [xmlblaster-devel] Client side persistence using SQLite
Avinash Atreya wrote:
We am looking at the XmlBlaster(XB) C client, more specifically at a
non-threaded client which offers client side persistence (to ensure that
messages are not lost).
This is what we use to compile the C client:
gcc -Wall -g -I. -I/usr/local/include/ -L/usr/local/lib -lsqlite \
-UXB_USE_PTHREADS -DXMLBLASTER_PERSISTENT_QUEUE_TEST=1 \
-o $1_HelloWorld HelloWorld_send.c \
util/helper.c util/msgUtil.c util/Properties.c \
socket/xmlBlasterSocket.c socket/XmlBlasterConnectionUnparsed.c \
(Helloworld_send.c is a simple program that we have written to publish
messages to the server)
Compiling with the above results in the errors:
socket/XmlBlasterConnectionUnparsed.c:348: structure has no member named
socket/XmlBlasterConnectionUnparsed.c:349: structure has no member named
socket/XmlBlasterConnectionUnparsed.c:365: structure has no member named
We checked QueueInterface.h and the structure QueueProperties does not
have any member named nodeId and XmlBlasterStruct no member named
userObject. Commenting these out results in successful compilation for
now as queueProperties->nodeId and queueProperties->userObject dont seem
to be used at least in the XmlBlasterConnectionUnparsed.c. (We am not
very familiar with the client side code yet and hence do not know about
there usage elsewhere)
Also, by the flow there, looks like the ordering of messages is not
preserved if a send fails. More specifically, if Msg#1 is persisted
because the connection is lost, and the connection is restored by the
time Msg#2 arrives, Msg#2 gets sent before Msg#1 though the client sent
the latter before. Is it the expected behaviour? We are looking Message
*Queues* functionality and given the same priority, FIFO needs to be
I couldnt find the code that takes messages out of the persistent
storage and sends it to the server again. (It has to be in the same
routine). I understand that this is still under test and you plan to add
that functionality. Is that so?
you are right, our SQLite persistence queue is not yet embedded to our C client
library. Currently it is only available embeded into the C++ client library.
My vision to implement it was as follows:
Add another layer 'XmlBlasterAccessFailSafe.c' which wraps around
'XmlBlasterAccessUnparsed.c'. This way the old code is not
touched and everybody can choose which features to use.
(A more component based approach instead of a layered approach would
probably be even more better - there you only add the features
you need as activated components, but i have not thought further about it).
The fail safe code needs to:
o Do ping the server if the connection is OK (optional)
o Do reconnect polling if the connection is down
o Queue all further published messages as we are in 'async mode'
In the Java and C++ client library we return a faked PublishReturnQos
so the client can continue.
The faked return has a state "QUEUED".
o On reconnect flush the queue (async mode)
- They are already sorted after their priority so you don't
need to do any priority specific coding!
- During flushing (it could be the ping thread) accept
further publishes which are put to queue!
o When the queue is empty switch to sync delivery of
new published messages.
See the slide show
The configuration parameters should be named similar to the C++
or Java client parameters, for example:
In a further step we may add to queue subscribe, unSubscribe and erase
requests as well, so the code should be open for such extensions.
XmlBlasterAccessUnparsed.c and XmlBlasterConnection.c should be free
of any queueing issues to be usable standalone as well (the queueing test code
in XmlBlasterAccessUnparsed.c will be removed, it was just a first try).
The C code must be tested at least on Windows (VC++) and Linux (gcc, icc)
and leak checked, the Solaris port i can do later myself on my Sun box.
A test case must be added to xmlBlaster/testsuite/src/c
and some documentation notes must be added to xmlBlaster/doc/requirements/client.c.xml
(similar to client.c.socket.xml and client.c.queue.xml).
Contributed code must follow the guidelines
and the license must be LGPL (probably with additional more free licenses like
Apache or BSD).
If you want to work on this task i will send you subversion write access,
we would open a branch in such a case.
We are looking forward for this extension,
The flow I am looking at is something like this:
begin Publish ( msg ...)
QoS = 0
pri = msg.priority
if connection not restored
update QoS //saying msg queued
if store is not empty
foreach p in 0..9 do //assume 0 is highest
msgArr = peek messages from the store with priority p
if p equals pri
push back msg into msgArr
if senddata(msgArr.....) is successful
remove all the peeked messages that were successfully sent
update QoS //indicating all the messages sent
if p equals pri
persist msg into store //we only peeked at the others, dint
update QoS //saying msg queue
else //store is empty
if senddata (msg... ) is successful
update QoS //saying msg sent
update QoS //saying msg queued
PublishArr will look something similar...
We would like to implement this unless we are duplicating any ongoing
efforts. Please let us know. We will be very happy to receive any
suggestions, information from your side.
Thanks in advance,