peerdirect distributed enterprise reach your technical peak

PeerDirect Distributed PeerDirect Distributed EnterpriseEnterprise

Reach Your Technical Peak

The Final Ascent

Session IV – Q4 2003

© 2003 Progress Software Corporation2Sim

plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

PeerDirect Distributed EnterprisePeerDirect Distributed Enterprise

■ Introduction to Database Replication

■ Architecture Overview

■ Configuration

■ Replication Rule Design

■ APIs and Event Callback

■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Introduction to Database ReplicationIntroduction to Database Replication

■ Back-up■ Fail-over■ Latency avoidance■ Load balancing■ Reporting■ Distributed environments

Why Database Replication?


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Define Distributed?

Data Center

Remote Offices

Remote Users

DB

Apps


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Goals of a Distributed Enterprise

■ Latency avoidance

■ Risk avoidance

■ Application functionality


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


DB

What if…

Data Center

Remote Offices

Remote Users

DB

Apps

Apps

Apps

Apps

DB

DB


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Breaks dependency on centralized application architectures

■ Radically improves employee productivity with decentralized applications– Disconnected use

■ Creates an inherently resilient application architecture

■ Eliminates the latency and availability problems caused by the centralized model

PeerDirect Distributed Enterprise


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Log based– Database activity logged for periodic ‘replay’ at all

locations■ Queue based

– Middleware intercepts application to database activity for periodic replay at all locations

■ Table based– Captures changes, queries source and duplicates

values at all locations

Approaches


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Log based– Log size– Can not remain dormant long– Periodically need to re-synch– Inefficient use of bandwidth– Typically only hub and spoke topology

■ Queue or Message based– All of the above– Application must be modified

■ Table based– Not real-time– Increases database size

Drawbacks of each approach


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Architecture OverviewArchitecture OverviewThe Basics

■ Table based– Vs Log or Queue based

■ System tables for storing metadata■ Engine appears as another user■ Access via SQL and ODBC


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Architecture OverviewArchitecture OverviewFeatures and benefits

Feature Benefit■ Bi-directional update-

everywhere model■ Read-write data between

multiple databases

■ Replicate, synchronize, and distribute corporate data across multiple locations


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Feature Benefit■ Scheduled

synchronization■ Net change compression■ Strong encryption■ Subset data using

‘slices’

■ Maximize network efficiency, reduce costs

■ Security– Controlled access to

data


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Feature Benefit■ Replicates between

different database types– MS SQL Server and

Progress– Oracle and PostreSQL

■ Supported databases– Progress

– Oracle

– SQL Server

– DB2

■ Share data in mixed environments

■ Consolidated view of corporate data


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Feature Benefit■ Multiple topologies

supported– Peer-to-peer– Hub and spoke– Load balance clustering

■ Provide new options for building scalable systems

■ Flexible configurations


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Feature Benefit■ Auto-discovery of nodes

within the replication network

■ Improve quality of service and system availability

■ Improved system administration

■ Allows mobile workers who are disconnected or have low-bandwidth limitations access to enterprise applications


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Feature Benefit■ Database level

configuration– Adds needed tables to

replicated database to manage replication

– Does not modify user-defined tables

■ Existing application does not necessarily need to be altered


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Architecture OverviewArchitecture Overview

■ Dynamic Data Slicing Architecture (DDSA)– Table partitioning - schema– Work set partitioning - query– Column level partitioning - fragment– Dynamic data migration

■ Dynamic Bandwidth-Managed Partner Selection (DBP)– Auto-discovery, auto-balanced– Avoids overloading any one server– Backbone, server and workstation algorithms

■ Collision Avoidance Methodology (CAM)– Record fragment management – related columns– Consistent resolution contracts– Rich API for custom resolution

Patented replication technology


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

System Tables41 tables ofsetup data

Control Tables1 table per

database table


Data Table “M” Rows

Database

Application

Control Table “M” Rows

PK

PK

Table-based


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Principal components

PeerDirect Distributed Enterprise

ReplicationEngine(PDRE)

ReplicationDesigner(PDRD)

ReplicationAdministrator

(PDRA)

PeerDirect Distributed Enterprise(PDDE)


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

ConfigurationConfiguration

■ Communicates via SQL through ODBC■ The closer the Replication Engine is to the

database, the better the performance– Supports Win32 and Linux

■ Configuration of InnerEdge and OuterEdge servers affect the way in which sites will be replicated

Replication Engine


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

ConfigurationConfigurationPDRE setup

Engine

ReplicateEngine

Replication

DB

DB

DB

DB

Engine

Replication

Engine

Replication

DB


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Site Types – 3 Types

■ Complete ■ High Availability■ Low Availability

– Based upon■ Data set■ Connectivity

– Determines replication partner selection

Site Types


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Considerations– A Replication Engine should be on either

end of any WAN connection– PDRE traffic is compressed and encrypted

whereas ODBC traffic is not– On a LAN, consider fault tolerance and

performance vs. cost of hardware– Engine may reside on a different platform

■ Win32 and Linux

Engine placement


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

ConfigurationsConfigurations

■ Dynamic partner selection algorithms allow for load balancing and fault tolerance

■ Partner selection is affected by a site’s type

■ Available site types– Low Availability– High Availability– Complete

Site types


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

ConfigurationsConfigurations

■ Low Availability– Sites that are mostly or occasionally

disconnected– Compact

■ High Availability– Sites that are mostly connected but do not

contain all data– ‘Spine sites’

Site types


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

ConfigurationConfigurationSimple Topologies – Inner Edge

– High Availability sites– All or most data slices– Reliable connectivity

Engine

ReplicateEngine

Replication

DB

Data Center


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


– High or Low availability sites– Pertinent data slices– Questionable connectivity

Engine

ReplicateEngine

Replication

DB

Remote Office

Simple Topologies – Outer Edge


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


Engine

ReplicateEngine

Replication

DB

Data Center

Engine

ReplicateEngine

Replication

DB

Engine

ReplicateEngine

Replication

DB

Remote Office

Engine

ReplicateEngine

Replication

DB

Remote Office

Sample PDDE Topology


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Aircraft Manufacturer– Aircraft maintenance application– Maintenance records were handled either on paper or in different

centralized databases– Maintenance issues cause commercial aircraft delays and military

readiness issues■ Solution: One PeerDirect InnerEdge Server and multiple OuterEdge

Workstations– Remote capabilities, work sets, and occasionally connected users– Aircraft maintenance data captured while aircraft is being maintained

resulting in significant cost savings

InnerEdge

Distributed Enterprise

OuterEdge

OuterEdge

OuterEdge

OuterEdge


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Medical Imaging– Patient test results contain images and complex data sets– Data only available at site where test performed– Collaboration with doctors at other locations nearly impossible

■ Solution: One PeerDirect InnerEdge Server and multiple OuterEdge Workstations– New test introduced into replication network– Can be viewed by physician at any location at anytime

InnerEdge

Distributed Enterprise

OuterEdge

OuterEdge

OuterEdge

OuterEdge


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Replication Rule DesignReplication Rule Design

■ Specify the application database■ Select the tables to replicate■ Define fragments to minimize the number of data

collisions■ Subset data into work sets■ Arrange tables in transaction sets and/or groups■ Verify and save rules■ Prepare to activate replication-enabled database

Basic steps to defining rules


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

Replication Rule DesignReplication Rule DesignAnalyze the schema Partial Sports2000

Schema

Order

PK,U1 Ordernum

FK1,FK2,FK3,U1 CustNumI1 OrderDate

ShipDatePromiseDateCarrierInstructionsPOTerms

I3 SalesRepFK3 BillToIDFK2 ShipToIDI2 OrderStatus

WarehouseNumCreditcard

RefCall

PK,U2,U1 CallNum

FK1,U1 CustNumCallDateSalesRep

U2 ParentI1 Txt

BillTo

PK,FK1 CustNumPK BillToID

NameAddressAddress2City

FK2 StatePostalCodeContactPhone

Salesrep

PK SalesRep

RepNameRegionMonthQuota

OrderLine

PK,FK1 OrdernumPK Linenum

FK2,I1 ItemnumPriceQtyDiscountExtendedPrice

I2 OrderLineStatus

ShipTo

PK,FK1 CustNumPK ShipToID

ContactAddressAddress2City

FK2 StatePostalCodePhoneCommentsName

Customer

PK CustNum

I3 NameAddressAddress2City

FK2 StateI2 Country

PhoneContact

FK1,I4 SalesRepI1 Comments

CreditLimitBalanceTermsDiscount

I2 PostalCodeFaxEmailAddress

Invoice

PK Invoicenum

FK2,I1 CustNumI2 InvoiceDate

AmountTotalPaidAdjustment

FK1,I3 OrderNumShipCharge

SalesRep Work Set


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Unit of replication is whole record■ Changes occur at multiple locations

– One user changes Addr– One user changes Limit

■ Can cause ‘False Collisions’

Generic replication

Name Addr City Zip Ctry Pymt Rating Limit


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Unit of replication is each field■ Changes occur at multiple locations

– One user changes Addr and Limit– One user changes Addr and Zip

■ Can cause ‘Silent Data Corruption’

Common solution to generic replication

Name Addr City Zip Ctry Pymt Rating LimitName Addr City Zip Ctry Pymt Rating Limit


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Unit of replication is a group of fields■ Changes occur at multiple locations

– One user changes Addr and Limit– One user changes Addr and Zip

■ Helps avoid ‘False Collisions’■ Optimizes the replication cycle

PeerDirect solution - fragments

Name Addr City Zip Ctry Pymt Rating Limit


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ PeerDirect allows you to define the business rules for sub-setting around base tables

Introduction to work sets

Cust

Prod

Price

HistCreditAcct

ADetl Trans

TDetl

Ctry

Off

HistCreditAcct

ADetl Trans

TDetl

Off Prod

Price


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Define the ‘Ctry’ base table■ Subscribe the site to its country

Defining work sets

142 - Bouchard

217 - W. Gates

330 - A. Palmer

401 - L. Haney

. . .

84006239

93050403

93072677

96193414

97005567

1 - Canada

2 - USA

3 - Australia

. . .

Ctry

Cust

Acct12 - New York

29 - Toronto

37 - Montreal

44 - Sydney

. . .

Off

550 - N. Peart

98717696

99656643

. . .

99700174


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Each site stores "all and only" the data it needs

■ Tables belonging to work sets are replicated based on a request, called a subscription

■ Nesting work sets allows for further sub-setting of data

Work sets


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ A set of insert, update or delete operations– A single unit of work– Must be completed as a whole

■ Offer the ability to enforce data integrity■ Two types of transactions supported

– Transaction sets– Transactions groups

Introduction to transactions


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness


■ Maintains transaction integrity of dependant table during incomplete replication session

■ Updates all dependant records in a single unit of work

■ Must include a base table■ Creation process similar to work sets

Transaction sets


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

APIs and Event CallbackAPIs and Event Callback

■ Available for use by all third-party developers

■ Available in both Unicode and ANSI■ 32-bit C-style function calls (standard

calling convention)

APIs


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

APIs and Event CallbackAPIs and Event Callback

■ Errors and Logging■ Control■ Command■ Configuration■ Database Record Information■ Event Handling■ Miscellaneous

API categories


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

APIs and Event CallbackAPIs and Event CallbackEvent handler callbacks

■ Allows you to modify some behavior■ Receive various kinds of event notifications■ PDRE events include

– Replication monitoring– Custom conflict resolution– Enhanced logging

■ Write and register custom DLLs■ Sample code

– C:\Program Files\PeerDirect\pdre\Samples\EvntClbck


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness




■ Configuration



■ Administration

Agenda


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

AdministrationAdministration

■ PDRE process listens for commands on port 2584– Network security infrastructure must be

configured to pass TCP data on this port■ ASCII protocol■ Provides interactive or programmatic

interface

Remote protocol


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

AdministrationAdministrationRemote protocol example


plify

you

r bu

sine

ssS

impl

ify y

our

busi

ness

AdministrationAdministration

■ Administers the replication network– Maintenance of site information– Subscribing and unsubscribing to and

from slices– Setting up replication schedules

■ Allows for remote administration of all tasks

GUI Administrator

peerdirect distributed enterprise reach your technical peak

Documents