SDR - Message Vrs Session Based Replication Print E-mail

Message-Based Replication

Many replication products use message-based replication (a.k.a. message queuing, store-and-forward replication), capturing changes into message packets which are then delivered using email, FTP, shared folder, or similar technologies.
The major concerns with message-based solutions are reliability, predictability and
efficiency :–

They are inherently fragile and administration intensive.
Message-based solutions can cause data discrepancies across replicated sites since they only transmit relative changes to data and never look directly at the actual data stored in each sites database. As a result, they occasionally require a manual ‘synchronisation’, which is a time consuming and complex process.

These site synchronisations incur significant cost as they often require DBA involvement on a regular basis.

Message-based solutions can also incur other administrative intensive tasks, such as the restoration of lost databases, and changes to the network topology such as adding a new database.

They are often inefficient, duplicating storage and increasing data transmission costs.
When a record is changed in a message-based system, each change can potentially be stored in different messages. By duplicating the data across messages, and not sending only the ‘net’ changes per replication, storage and transmission costs can be increased greatly.

In addition, many systems send complete row changes, when only part of a row may have changed. Again, resulting in increased storage and transmission costs.

They are unable to offer timely and robust conflict detection and resolution.
Conflicts are detected at remote sites as messages flow asynchronously between them. As such, conflicts and their resolutions can routinely ‘chase’ between sites, until the issue is resolved.

They have inherent points of failure, and scale poorly.
Many message-based solutions are based on a hub and spoke topology.

This type of topology introduces a single point of failure, the hub. Scalability is limited, as the central hub becomes the bottleneck of the distributed system.

Since message-based solutions often rely upon another transport mechanism such as email or FTP they are unable to offer reliable delivery time guarantees.

Session-Based Replication using SDR

Session-based replication (a.k.a. point-to-point, connection-based) performs over a network connection between two databases, both of which actively participate in the replication session and simultaneously send and receive changes.

The main advantages are :–

Optimal data transmission efficiency.
Even if the same data is changed many times between replication sessions, only the final data, or true net change, is transmitted during the replication session. This avoids redundantly transmitting multiple versions of the same record.

There is no needless data re-transmission during session failure, since the replication engine only synchronises exactly the changes that remain.

Timely and robust conflict detection and resolution.
Conflicts are detected and resolved simultaneously by both sites whilst participating in a session.

Ease of use, reliability and scalability.
Connection-based replication is less costly to support because it is less complex for users and administrators, and has fewer points of failure.

It is not limited to hub-and-spoke configurations, and can allow true peer-to-peer replication, which includes load balancing, replication failover, and dynamic rerouting when sites are unavailable.

Data Transmission, Performance and Efficiency

The SDR solutions is a ‘direct to database’ replication tool that only transmits the net changes for a given record. It does not store duplicated data across messages.

The SDR solution views the records, not as an indivisible block or as a collection of loose unrelated fields, but as a set of ‘record fragments’. For example, fields within an address such as the postcode and city may be treated as one fragment, with the telephone being treated as another.

With the fragments being ‘the unit of replication’, only the small parts of a record, not full records, need to be replicated, greatly reducing the amount of data that is transmitted.

Replication session overhead is minimal, typically adding less than 10k of data per session, the rest of the session is devoted to transmitting data. If the replication session is interrupted, only the remaining data is transferred.

Data transmissions are automatically compressed (typical data transmissions are reduced by a factor of 4 to 1) and then encrypted. This efficiency allows for replication across network connections that are often slow and expensive, including satellite, mobile phone, and dial-up modem connections.

For each record fragment, SDR also stores a cryptographic integrity check for the data in the fragment. During replication, if a fragments is found to be tampered or corrupted, full information is written to the local log, and if possible the fragment’s data is automatically replicated from another site that has a valid copy of the fragment.

Industry standard, government-endorsed security is used by SDR; Triple-DES, SHA, and ECC security algorithms. This makes is safe to replicate over public networks such as the Internet, and wireless networks.

Conflict Resolution, Integrity and Reliability

False Update Collisions
If two separate users simultaneously change different fields of the same record, at different sites, and the changes are not directly related, both changes should survive. For example, a user updates the first line of an address at one site, whilst another user amends the telephone number of the same address at different site.

The problem with most systems is that they will incorrectly flag this as a collision, because they replicate at record-level. This is a serious integrity weakness if only one of the updates survives.

Resolving such conflicts can be a major source of administrative overhead.

The ‘fragments’ feature within SDR correctly allows updates to different fields in the same record at different sites without any false collision.

Some replication products do ‘field-level’ replication, and so updated to different fields of the same record at different sites are always allowed. This is a serious integrity issues if the some fields should not change independently, e.g. customer address field, and customer zip code field. This can provide a false sense of security, which can lead to data corruption.

True Update Collisions
A true collision occurs when users change information that is related. For example, a user updates the first line of an address at one site, and another user updates the city of the same address at another site.

In this case, a true collision, only one change should be allowed to persist.

In SDR, all such collisions are locally logged, and replicated, allowing administrators at remotes sites to inspect collisions throughout the distributed system.

When a true collision or conflict does occur, the default behaviour is ‘most recent update wins’ based on time, although the default can be changed, and customised.

A notification can also be configured, so that administrator or an automated program can inspect, and if necessary, resolve the competing updates.

Ease of Use, Reliability and Scalability
With the SDR system, distribution rules are configured once, at design time. All sites where a database is installed use the same shared rules.

There is no need to fix the network topology of the replication network, and changes to the topology (such as adding new sites) do not require time-consuming re-configuration and re-synchronisation.

The SDR architecture allows the entire distributed database system to be administered from any site, and an authorized administrator my set replication schedules, data subsets (“slices”), subscriptions and any other configuration from any location.

As the SDR system uses a separate thread for each replication session. It can run on a small notebook computer but scale to multiprocessor servers, and take advantage of the extra power.

Having the ability to replicate with many sites simultaneously, concurrent sessions can ‘follow’ each other through the schema, gathering and applying updates. Most replication products are limited to a single replication session.

This concurrent ability enables highly efficient ‘pass-through’ replication. For example, site B can start replication with site A, and whilst replication is in progress, site C can also start replication with site A. A replication session overlap occurs and updates travel from site B to site C nearly as quick as if site B and C were replicating directly.

With the SDR solution offering dynamic configuration and automatic load balancing, more advanced topologies, such as mesh, are available. This results in automatic replication failover if some sites are temporarily or permanently unavailable.

Since there is no central server restriction, it is possible to support tiered hierarchies of sites, allowing a group of central servers to replicate with a lower tier, such as the leaf sites.

Open System, Heterogeneous, Standards-Based.
The SDR solution supports Oracle, Microsoft SQL Server, Informix, Sybase ASE, Sybase SQL Anywhere, Progress, DB2 and PostgreSQL.
As the SDR system integrates as a database client, it is able to replicate high-end databases regardless of operating system platform, e.g. Windows, Unix, or other.

Related Articles: