[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster-devel] Client side persistence using SQLite

Avinash Atreya wrote:
Hi,
We am looking at the XmlBlaster(XB) C client, more specifically at a non-threaded client which offers client side persistence (to ensure that messages are not lost).

This is what we use to compile the C client:
gcc -Wall -g -I. -I/usr/local/include/ -L/usr/local/lib -lsqlite \
-UXB_USE_PTHREADS -DXMLBLASTER_PERSISTENT_QUEUE_TEST=1 \
-o $1_HelloWorld HelloWorld_send.c \
util/helper.c util/msgUtil.c util/Properties.c \
socket/xmlBlasterSocket.c socket/XmlBlasterConnectionUnparsed.c \
util/queue/SQLiteQueue.c

(Helloworld_send.c is a simple program that we have written to publish messages to the server)

Compiling with the above results in the errors:
socket/XmlBlasterConnectionUnparsed.c:348: structure has no member named `nodeId'
socket/XmlBlasterConnectionUnparsed.c:349: structure has no member named `nodeId'
socket/XmlBlasterConnectionUnparsed.c:365: structure has no member named `userObject'

We checked QueueInterface.h and the structure QueueProperties does not have any member named nodeId and XmlBlasterStruct no member named userObject. Commenting these out results in successful compilation for now as queueProperties->nodeId and queueProperties->userObject dont seem to be used at least in the XmlBlasterConnectionUnparsed.c. (We am not very familiar with the client side code yet and hence do not know about there usage elsewhere)

Also, by the flow there, looks like the ordering of messages is not preserved if a send fails. More specifically, if Msg#1 is persisted because the connection is lost, and the connection is restored by the time Msg#2 arrives, Msg#2 gets sent before Msg#1 though the client sent the latter before. Is it the expected behaviour? We are looking Message *Queues* functionality and given the same priority, FIFO needs to be guaranteed.

I couldnt find the code that takes messages out of the persistent storage and sends it to the server again. (It has to be in the same routine). I understand that this is still under test and you plan to add that functionality. Is that so? 

Hi Avinash,

you are right, our SQLite persistence queue is not yet embedded to our C client
library. Currently it is only available embeded into the C++ client library.

My vision to implement it was as follows:

Add another layer 'XmlBlasterAccessFailSafe.c' which wraps around
'XmlBlasterAccessUnparsed.c'. This way the old code is not
touched and everybody can choose which features to use.
(A more component based approach instead of a layered approach would
probably be even more better - there you only add the features
you need as activated components, but i have not thought further about it).

The fail safe code needs to:

o Do ping the server if the connection is OK (optional)

o Do reconnect polling if the connection is down

o Queue all further published messages as we are in 'async mode'
  In the Java and C++ client library we return a faked PublishReturnQos
  so the client can continue.
  The faked return has a state "QUEUED".

o On reconnect flush the queue (async mode)
  - They are already sorted after their priority so you don't
    need to do any priority specific coding!
  - During flushing (it could be the ping thread) accept
    further publishes which are put to queue!

o When the queue is empty switch to sync delivery of
  new published messages.


See the slide show

  http://www.xmlblaster.org/soffice_backup/PubSubWithQueue/PubSubWithQueue.html


NOTES:

The configuration parameters should be named similar to the C++
or Java client parameters, for example:

 -dispatch/connection/pingInterval

In a further step we may add to queue subscribe, unSubscribe and erase
requests as well, so the code should be open for such extensions.

XmlBlasterAccessUnparsed.c and XmlBlasterConnection.c should be free
of any queueing issues to be usable standalone as well (the queueing test code
in XmlBlasterAccessUnparsed.c will be removed, it was just a first try).

The C code must be tested at least on Windows (VC++) and Linux (gcc, icc)
and leak checked, the Solaris port i can do later myself on my Sun box.
A test case must be added to xmlBlaster/testsuite/src/c
and some documentation notes must be added to xmlBlaster/doc/requirements/client.c.xml
(similar to client.c.socket.xml and client.c.queue.xml).
Contributed code must follow the guidelines

 http://www.xmlblaster.org/xmlBlaster/doc/howto/developer-guidelines.html

and the license must be LGPL (probably with additional more free licenses like
Apache or BSD).

If you want to work on this task i will send you subversion write access,
we would open a branch in such a case.

We are looking forward for this extension,

best regards

Marcel



The flow I am looking at is something like this:

begin Publish ( msg ...)
QoS = 0
pri = msg.priority
if connection not restored
persist msg
update QoS //saying msg queued
else
if store is not empty
foreach p in 0..9 do //assume 0 is highest
msgArr = peek messages from the store with priority p
if p equals pri
push back msg into msgArr
endif
if senddata(msgArr.....) is successful
remove all the peeked messages that were successfully sent
update QoS //indicating all the messages sent
else
if p equals pri
persist msg into store //we only peeked at the others, dint remove them
update QoS //saying msg queue
endif
endif
done
else //store is empty
if senddata (msg... ) is successful
update QoS //saying msg sent
else
persist msg
update QoS //saying msg queued
endif
endif
endif
return QoS
end

PublishArr will look something similar...

We would like to implement this unless we are duplicating any ongoing efforts. Please let us know. We will be very happy to receive any suggestions, information from your side.

Thanks in advance,
Avinash







--
http://www.xmlBlaster.org