[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster] ssl and compression with SOCKET

Hi Marcel!

this is a useful extension to our basic SOCKET implementation.
If you are willing to donate it under LGPL license we would be
happy to add it to the xmlBlaster distribution.

I'm willing.

If so, you can send the patch directly to my mail address.
You would need to add a test case as well and some documentation in
the SOCKET requirement (xmlBlaster/doc/requirements/protocol.socket.xml).

Is there a compatible C compression library with a free license around
which could be added to the C/C++ client library?
Do you have a property for a minimum message size to switch on

Yes, there is a compatible C compression library. I mentioned that jzlib was used as the java library. As it is based on (actually, quite copy-paste work) the GNU zlib C library, the libs and their inputs/outputs have to be fully compatible. It's also stated on their website: http://www.jcraft.com/jzlib/

Let me explain the implementation details:
- Currently, compression is turned on if and only if SSL sockets are used. However, these features can be separated easily.
- The _whole_ tcp stream is compressed, because it was the easiest to implement it this way. This means that the "compression window" is not reinitialized on every message; The dictionary isn't flushed in the deflater, so repeating sequences are compressed even if they were in a previous message. I thought that would enable the deflater to compress the beginning of the message, too, and helps to achive better results when the same type of small messages are sent frequently (in my case, that is very true).
What I don't know is how all this behaves when random messages are sent on a connection. I mean, what if virtually every message would have some ugly binary content, for example? I still believe this would not make the messages bigger on average.

A deeper understanding (and more modification) of the SOCKET implementation would be required if I wanted to compress individual messages. This is not impossible, either. Moreover, if I put the compression filter in the right place, we would have compression for every protocol, wouldn't we?

Thinking about this, I already have another implementation which, in a way, supports compressing individual messages: it only compresses a block of bytes when the stream is flushed or when its buffer is full. When xmlBlaster sends a message, it writes the whole message to the OutputStream, then flushes it. When flush is invoked, the buffer state would enable it to decide whether to compress the current buffer (which should be a message, or a big part of it) or not. This way, we would lose the state of the deflater object and start with a new one on every flush, compress the buffer, and see if it's smaller than the original buffer. If it is, we'd send it compressed, otherwise not. Some simple protocol wrapper is also needed, but that is no problem.

I think these are our options.

I will assemble a patch for you as soon as I sepatated the SSL and compression layer and cleaned up some code. If you like the idea of deciding compression on a per-flush basis, then that will be included, too.