xmlBlaster - Requirement contrib.replication

REQUIREMENT

contrib.replication

Type NEW

Priority HIGH

Status CLOSED

Topic Replication is a plugin framework used to replicate data between database (RDBMS) sources and sinks.

Des
cription

Replication Overview
The Replication framework makes use of the DbWatcher- and the DbWriter frameworks. Its goal is to offer the xmlBlaster user a way to replicate data from one or several source databases to one or more sink databases. The framework has been designed for a singlemaster multislave approach. What does that mean ? It means that it does not offer any conflict resolution mechanism. In other words it is safe for cases where you have one master database, also called the source for the replication, and one or more slaves, also called the sinks of the replication or replicas. It will not be safe where you have more than one master. There are situations where this framework is currently used with more than one source for one sink, but there are limitations, for example you must ensure that the same data (at least on a row granularity) always comes from one single source.

replication with xmlBlaster

Which features does this framework offer ?

Single Master / Multi Slave approach

Conventional one to one Replication

Replication with Data Manipulation on a per sink individual basis with no intervention on the master side

Replication with Data Filtration on a per sink individual basis with no intervention on the master side

Use of all Communication Protocols offered by XmlBlaster (even the most fancy ones)

Persistence of Messages, Failover and guaranteed sequence of messages

Transactions are handled atomically and sent in a single Message

Possibility to interchange this with other sources/sinks for for example Meta-Databases

Plugin approach to allow business-specific functionality

Possibility to intermix different database versions and vendors

Performance tuning for specific vendors (for example for initial update)

Highly Compressed communication

Encrypted and secure replication

Unlimited number of sinks

Efficient and reliable even on unreliable networks

Why yet another replication solution when there are so many out there already ?
Because it is difficult (or impossible) to find a replication software which can offer all of the following:

Replicate between different databases (different versions or different vendors)

Allow individual (specific for every sink) business specific manipulation or filtering

pass complex and high secure gateways or firewalls

open source

replicated is xml based and can be used directly on an application level

...

Since the framework is highly flexible, it is also quite complex. Probably the simplest way to describe it is with an easy tutorial. I will start with a very simple use case. Parallely a description of what is going on behind the scenes will also be given. Prerequisistes for this tutorial is that you have an oracle database up and running.

The demo is also available as a Flash movie here.

Prepare, build and start the server

To avoid messy and strange behaviours make sure you don't have any old configuration around. Consider that as cleaning up the kitchen before starting to cook a good meal, and believe me, this meal is really good!
You can clean up by changing to your home directory and delete xmlBlaster.properties and xmlBlasterPlugins.xml

Delete old peristent queues. These normally reside in your ${HOME}/tmp directory. It normally suffices to delete all xmlBlaster* and client* files.

open the ${HOME}/xmlBlasterPlugins.xml file and uncomment the MX4JAdaptorPlugin, this way it will become active.

Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with . .bashrc).

Start the xmlBlaster with JMX enabled:
java org.xmlBlaster.Main -xmlBlaster/jmx/HtmlAdaptor true

Start the Demo Client for the Replication

Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with . .bashrc).

Go to the directory where the demo resides: ${XMLBLASTER_HOME}/demo/javaclients/contrib/replication.

Put in this directory the jdbc driver you want to use. Currently we only have tested with Oracle, so copy the latest driver (we used ojdbc14.jar) and put it in this directory.

Add this directory and the jar file to the CLASSPATH (unixes: export CLASSPATH=.:ojdbc14.jar:${CLASSPATH}).

Edit the first three lines of the files in the current directory readerDemo1.properties and writerDemo1.properties to make them correspond to your database. The settings are currently pointing to an oracle database on the localhost on the default port with a SID=TEST, and a user (or schema) XMLBLASTER

Now you will start the client. The client is composed by a master (the reader of the database or source), and a slave (which is the destination of the Replication). Internally xmlBlaster uses here a DbWatcher and a DbWriter. Invoke:
java javaclients.contrib.replication.ReplDemo -master readerDemo1.properties -slave writerDemo1.properties -interactive true
You will see the client starts and both the reader and the writer connect to the xmlBlaster. At the end you will see the text REPLICATION AGENT IS NOW READY and a prompt appears. If you press ENTER, then a new line appears. On the line you could now execute SQL Statements. The prompt tells you on which Database instance the SQL Statement will be executed. Here you see master telling you the statements will be executed on the master and therefore will be replicated. If you would have started only the slave, the prompt slave would have appeared. Of course for the sake of simplicity we are using one single database here.

In the configuration file you defined the tables to be replicated:
table.${db.user}.DEMO_USERS=trigger=DEMO_USERS_TRG
Meaning DEMO_USERS and DEMO_ACCOUNTS will be replicated, of course if and only if they exist. You also can see the trigger name used to detect changes on the DEMO_USERS will be called DEMO_USERS_TRG. Not choosing anything here would let the application decide a name for it.

Now you can add the table DEMO_USERS by typing on the command line of the started client:
create table demo_users (account varchar(100) not null, name varchar(100), firstname varchar(100), primary key(account))
When completed press ENTER.

Once the table has been created it is time to populate it with some data:
insert into demo_users values ('account1', 'Topo', 'Gigio') insert into demo_users values ('account2', 'Mister', 'Bean')

Start the JMX Monitor to initiate the Replication
Open an html browser of your choice, for example Firefox. Point it to http://localhost:9999. You will see something like the following:

start replication

The Source (here DEMO_) is the prefix identifying the source of the replication. The Destination is the name identifying the Destination of the replication. In this demo you only have one possible combination allowed. So click on the Initiate Repl. Button. This way the replication will start.

By clicking on the button you will be redirected to another page displaying all initiated and running replications. In this case you only have one single entry.

replication list

Note that the status is initially switching from blue to yellow. When it is in the blue phase (INITIAL), the source is prepared for the replication. During this phase the Active/Standby Status is in Standby. This means the replication is temporarly hold back, no message is sent to the destination.
After a very short time the status switches to blue (TRANSITION). During this phase the source is ready and the messages are delivered to the destination but the destination is not completely up to date yet. You will see that during this phase the Active/Standby flag is on Active meaning processed messages are delivered immediately.

When all initial messages have been delivered, the status is switching again, this time from yellow to green (RUNNING). From now on the destination is ready and the initial data have been replicated.

On the server logs you will see that the initial command has been executed and on the client logs you will see the PL/SQL Code of the trigger followed by a sequence of two XmlBlaster Messages each in xml format. The first of them is the CREATE statement itself and the second is the message containing all initial data of the replicated table.

You can now verify that the new table DEMO_USERS_REPLICA really has been created and that it contains the same entries as the source by invoking on the prompt:
select * from demo_users_replica
This way you replicated data from one table to another.

Configure
NOTE: Configuration parameters are specified on command line (-someValue 17) or in the xmlBlaster.properties file (someValue=17). See requirement "util.property" for details.
Columns named Impl tells you if the feature is implemented.
Columns named Hot tells you if the configuration is changeable in hot operation.

See API org.xmlBlaster.contrib.dbwriter.DbWriter

See API org.xmlBlaster.contrib.dbwatcher.DbWatcher

See API org.xmlBlaster.contrib.replication.ReplicationAgent

See API org.xmlBlaster.contrib.replication.ReplSlave

See API org.xmlBlaster.contrib.replication.impl.ReplManagerPlugin

See Replication Documentation

See REQ contrib.dbwatcher

See REQ contrib.dbwriter

See TEST org.xmlBlaster.test.contrib.dbwriter.TestRecordParsing

This page is generated from the requirement XML file xmlBlaster/doc/requirements/contrib.replication.xml

Back to overview

Topic	Replication is a plugin framework used to replicate data between database (RDBMS) sources and sinks.
Des cription	Replication Overview The Replication framework makes use of the DbWatcher- and the DbWriter frameworks. Its goal is to offer the xmlBlaster user a way to replicate data from one or several source databases to one or more sink databases. The framework has been designed for a singlemaster multislave approach. What does that mean ? It means that it does not offer any conflict resolution mechanism. In other words it is safe for cases where you have one master database, also called the source for the replication, and one or more slaves, also called the sinks of the replication or replicas. It will not be safe where you have more than one master. There are situations where this framework is currently used with more than one source for one sink, but there are limitations, for example you must ensure that the same data (at least on a row granularity) always comes from one single source. Which features does this framework offer ? Single Master / Multi Slave approach Conventional one to one Replication Replication with Data Manipulation on a per sink individual basis with no intervention on the master side Replication with Data Filtration on a per sink individual basis with no intervention on the master side Use of all Communication Protocols offered by XmlBlaster (even the most fancy ones) Persistence of Messages, Failover and guaranteed sequence of messages Transactions are handled atomically and sent in a single Message Possibility to interchange this with other sources/sinks for for example Meta-Databases Plugin approach to allow business-specific functionality Possibility to intermix different database versions and vendors Performance tuning for specific vendors (for example for initial update) Highly Compressed communication Encrypted and secure replication Unlimited number of sinks Efficient and reliable even on unreliable networks Why yet another replication solution when there are so many out there already ? Because it is difficult (or impossible) to find a replication software which can offer all of the following: Replicate between different databases (different versions or different vendors) Allow individual (specific for every sink) business specific manipulation or filtering pass complex and high secure gateways or firewalls open source replicated is xml based and can be used directly on an application level ... Since the framework is highly flexible, it is also quite complex. Probably the simplest way to describe it is with an easy tutorial. I will start with a very simple use case. Parallely a description of what is going on behind the scenes will also be given. Prerequisistes for this tutorial is that you have an oracle database up and running. The demo is also available as a Flash movie here. Prepare, build and start the server To avoid messy and strange behaviours make sure you don't have any old configuration around. Consider that as cleaning up the kitchen before starting to cook a good meal, and believe me, this meal is really good! You can clean up by changing to your home directory and delete `xmlBlaster.properties` and `xmlBlasterPlugins.xml` Delete old peristent queues. These normally reside in your `${HOME}/tmp` directory. It normally suffices to delete all `xmlBlaster` and `client` files. open the ${HOME}/xmlBlasterPlugins.xml file and uncomment the MX4JAdaptorPlugin, this way it will become active. Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with `. .bashrc`). Start the xmlBlaster with JMX enabled: java org.xmlBlaster.Main -xmlBlaster/jmx/HtmlAdaptor true Start the Demo Client for the Replication Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with `. .bashrc`). Go to the directory where the demo resides: `${XMLBLASTER_HOME}/demo/javaclients/contrib/replication`. Put in this directory the jdbc driver you want to use. Currently we only have tested with Oracle, so copy the latest driver (we used `ojdbc14.jar`) and put it in this directory. Add this directory and the jar file to the CLASSPATH (unixes: `export CLASSPATH=.:ojdbc14.jar:${CLASSPATH}`). Edit the first three lines of the files in the current directory `readerDemo1.properties` and `writerDemo1.properties` to make them correspond to your database. The settings are currently pointing to an oracle database on the localhost on the default port with a SID=TEST, and a user (or schema) XMLBLASTER Now you will start the client. The client is composed by a master (the reader of the database or source), and a slave (which is the destination of the Replication). Internally xmlBlaster uses here a DbWatcher and a DbWriter. Invoke: java javaclients.contrib.replication.ReplDemo -master readerDemo1.properties -slave writerDemo1.properties -interactive true You will see the client starts and both the reader and the writer connect to the xmlBlaster. At the end you will see the text REPLICATION AGENT IS NOW READY and a prompt appears. If you press `ENTER`, then a new line appears. On the line you could now execute SQL Statements. The prompt tells you on which Database instance the SQL Statement will be executed. Here you see master telling you the statements will be executed on the master and therefore will be replicated. If you would have started only the slave, the prompt slave would have appeared. Of course for the sake of simplicity we are using one single database here. In the configuration file you defined the tables to be replicated: table.${db.user}.DEMO_USERS=trigger=DEMO_USERS_TRG Meaning DEMO_USERS and DEMO_ACCOUNTS will be replicated, of course if and only if they exist. You also can see the trigger name used to detect changes on the DEMO_USERS will be called DEMO_USERS_TRG. Not choosing anything here would let the application decide a name for it. Now you can add the table DEMO_USERS by typing on the command line of the started client: create table demo_users (account varchar(100) not null, name varchar(100), firstname varchar(100), primary key(account)) When completed press `ENTER`. Once the table has been created it is time to populate it with some data: insert into demo_users values ('account1', 'Topo', 'Gigio') insert into demo_users values ('account2', 'Mister', 'Bean') Start the JMX Monitor to initiate the Replication Open an html browser of your choice, for example Firefox. Point it to http://localhost:9999. You will see something like the following: The Source (here DEMO_) is the prefix identifying the source of the replication. The Destination is the name identifying the Destination of the replication. In this demo you only have one possible combination allowed. So click on the Initiate Repl. Button. This way the replication will start. By clicking on the button you will be redirected to another page displaying all initiated and running replications. In this case you only have one single entry. Note that the status is initially switching from blue to yellow. When it is in the blue phase (INITIAL), the source is prepared for the replication. During this phase the Active/Standby Status is in Standby. This means the replication is temporarly hold back, no message is sent to the destination. After a very short time the status switches to blue (TRANSITION). During this phase the source is ready and the messages are delivered to the destination but the destination is not completely up to date yet. You will see that during this phase the Active/Standby flag is on Active meaning processed messages are delivered immediately. When all initial messages have been delivered, the status is switching again, this time from yellow to green (RUNNING). From now on the destination is ready and the initial data have been replicated. On the server logs you will see that the initial command has been executed and on the client logs you will see the PL/SQL Code of the trigger followed by a sequence of two XmlBlaster Messages each in xml format. The first of them is the CREATE statement itself and the second is the message containing all initial data of the replicated table. You can now verify that the new table DEMO_USERS_REPLICA really has been created and that it contains the same entries as the source by invoking on the prompt: select * from demo_users_replica This way you replicated data from one table to another.
Configure	NOTE: Configuration parameters are specified on command line (-someValue 17) or in the xmlBlaster.properties file (someValue=17). See requirement "util.property" for details. Columns named Impl tells you if the feature is implemented. Columns named Hot tells you if the configuration is changeable in hot operation.
See API	org.xmlBlaster.contrib.dbwriter.DbWriter
See API	org.xmlBlaster.contrib.dbwatcher.DbWatcher
See API	org.xmlBlaster.contrib.replication.ReplicationAgent
See API	org.xmlBlaster.contrib.replication.ReplSlave
See API	org.xmlBlaster.contrib.replication.impl.ReplManagerPlugin
See	Replication Documentation
See REQ	contrib.dbwatcher
See REQ	contrib.dbwriter
See TEST	org.xmlBlaster.test.contrib.dbwriter.TestRecordParsing