experiences and lessons learned 6-node active-active oracle goldengate luke davies

Experiences and Lessons Learned

6-Node Active-Active Oracle GoldenGate

Luke Davies

Why Pythian• Recognized Leader:

• Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration

• Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:

• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 8 Oracle ACEs/ACE Directors

• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC

• Global Reach & Scalability:

• 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response

About Luke Davies• Team Technical Lead• Oracle DBA since V6 (1990)• BSc Microbiology• With Pythian since 2008• Main Focus

• Oracle Core• Oracle Performance Tuning

Set Up

Set Up• Business requirements

• High availability

• Easily Scalable

• Application releases with zero downtime

• Database patches/upgrades with zero downtime

• Real-time reporting database with no impact to Online users

• Other application entry points with no impact to Online users

• Original Solution based on • Oracle Standard Edition 10gR2

• GoldenGate V10.4

Set Up

Experiences and Lessons Learned

Topics• Active/Active considerations

• Supplemental Logging• DDL Replication• Conflict Avoidance• Conflict Detection• Release Management• Data comparison

• General considerations• Trail file deletion• Archive log deletion

Supplemental Logging• Required at the database level (Minimum Level)

• ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

• Minimal Supplemental Logging

“logs the minimal amount of information needed for LogMiner to identify, group, and merge the redo operations associated with DML changes”

• Identification key logging (PK, UK, FK)• Table Level • Database Level• Method

• SQL e.g. ALTER TABLE <> ADD SUPPLEMENTAL LOG DATA …

• GoldenGate e.g. ADD TRANDATA <Table Name>

• Cost of Omission

Changes may fail

Supplemental LoggingSource Tg

create table repuser2.tbl4 (id number, fname varchar2(100), lname varchar2(100), issued date);

2012-11-14 09:20:57 INFO OGG-00487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL operation included [INCLUDE ALL OBJNAME REPUSER2.*], optype [CREATE], objtype [TABLE], objowner [REPUSER2], objname [TBL4].

2012-11-14 09:20:57 INFO OGG-00497 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Writing DDL operation to extract trail file.

2012-11-14 09:21:03 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGT_61642" ("ID","FNAME","LNAME","ISSUED") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 141)].

2012-11-14 09:21:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

2012-11-14 09:21:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [CREATE] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG GROUP "GGT_61642" /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

select * from dba_log_groups where table_name='TBL4';

OWNER LOG_GROUP_NAME TABLE_NAME LOG_GROUP_TYPE ALWAYS GENERATED----- -------------- ---------- -------------- ------ ---------REPUSER2 GGS_61642 TBL4 USER LOG GROUP ALWAYS USER NAME

Drop SL for all namedcolumn

Supplemental LoggingSrc Target

2012-11-14 09:21:05 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation [create table repuser2.tbl4 (id number, fname varchar2(100), lname varchar2(100), issued date) (size 95)].

2012-11-14 09:21:05 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of

mapped scope, after mapping new operation [create table REPUSER2.TBL4 (id number, fname varchar2(100), lname varchar2(100), issued date) (size 95)].

2012-11-14 09:21:05 INFO OGG-01407 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Setting current schema for DDL operation to [SYS].

2012-11-14 09:21:05 INFO OGG-00484 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Executing DDL operation.

2012-11-14 09:21:06 INFO OGG-00483 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL operation successful.

2012-11-14 09:21:06 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggsdb].

desc repuser2.tbl4

Name Null? TypeID NUMBERFNAME VARCHAR2(100)LNAME VARCHAR2(100)ISSUED DATE

No SLIn

Target

TBL4in

Target

Supplemental LoggingSource Tg

alter table repuser2.tbl4 add primary key(id);

2012-11-14 09:51:07 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation [alter table repuser2.tbl4 add primary key(id) (size 47)], start SCN [9690796], commit SCN [9690813] instance [orcl1123 (1)], DDL seqno [1008], marker seqno [1008].

2012-11-14 09:51:07 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGT_61642" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 116)].

2012-11-14 09:51:07 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [ALTER] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

2012-11-14 09:51:07 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN DATA for table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" ADD SUPPLEMENTAL LOG GROUP "GGS_61642" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 116)].

2012-11-14 09:51:07 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN DATA for [ALTER] DDL operation, table [REPUSER2.TBL4], operation [ALTER TABLE "REPUSER2"."TBL4" DROP SUPPLEMENTAL LOG GROUP "GGT_61642" /* GOLDENGATE_DDL_REPLICATION */ (size 103)].

select * from dba_log_group_columns where table_name='TBL4';

OWNER LOG_GROUP_NAME TABLE_NAME COLUMN_NAME POSITION LOGGING_PROPERTY----- -------------- ---------- ----------- -------- ----------------REPUSER2 GGS_61642 TBL4 ID 1 LOG

SL for all

columns is

dropped

SL for IDis

added(PK)

Supplemental LoggingSource

Target

2012-11-14 09:51:11 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL

found, operation [alter table repuser2.tbl4 add primary key(id) (size 47)].

mapped scope, after mapping new operation [alter table REPUSER2.TBL4 add primary key(id) (size 47)].

no rows

NO SLIn

TargetYET !!

Supplemental Logging

T Source Target

t1 insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

What if we keep running replication this way

T Source Target

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------ ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

T Source Target

t2 update repuser2.tbl4 set lname='TestL4' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

T Source Target

t2 UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0OCI Error ORA-01403: no data found, SQL <UPDATE "REPUSER2"."TBL4" SET "LNAME" = :a1 WHERE "ID" = :b0>Operation failed at seqno 5 rba 4598Discarding record on action DISCARD on error 1403Problem replicating REPUSER2.TBL4 to REPUSER2.TBL4Record not foundMapping problem with compressed update record (target format)...*ID =LNAME = TestL4

update repuser2.tbl4 set lname='TestL4' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

MissingValue!

T Source Target

insert into repuser2.tbl4 values (1,'Test','TestL',sysdate);insert into repuser2.tbl4 values (2,'Test','TestL2',sysdate);insert into repuser2.tbl4 values (3,'Test','TestL3',sysdate);

What if supplemental logging is enabled on target

T Source Target

add trandata repuser2.tbl4

T Source Target

update repuser2.tbl4 set lname='TestLB' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLB 14/11/12

T Source Target

UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0a1 : TestLBb0 : 3

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12 AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12 AAAPDKAAEAAAAIOAAC 3 Test TestLB 14/11/12

update repuser2.tbl4 set lname='TestLB' where fname='Test' and rowid='AAAar/AAEAAAE5HAAC‘;

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLB 14/11/12

T Source Target

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- --------

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestL3 14/11/12

What if supplemental logging is enabled on DB level

T Source Target

delete trandata repuser2.tbl4alter database add supplemental log data (primary key) columns;alter system switch logfile;

ROWID ID FNAME LNAME ISSUED------------------ -- ----- ------- ---------

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestL3 14/11/12

T Source Target

update repuser2.tbl4 set lname='TestLC’ where fname='Test' and rowid=‘AAAar/AAEAAAE5HAAC’;

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLC 14/11/12

T Source Target

UPDATE "REPUSER2"."TBL4" SET "LNAME"=:a1 WHERE "ID"=:b0a1 : TestLCb0 : 3

AAAPDKAAEAAAAIOAAA 1 Test TestL 14/11/12AAAPDKAAEAAAAIOAAB 2 Test TestL2 14/11/12AAAPDKAAEAAAAIOAAC 3 Test TestLC 14/11/12

update repuser2.tbl4 set lname='TestLC’ where fname='Test' and rowid=‘AAAar/AAEAAAE5HAAC’;

AAAar/AAEAAAE5HAAA 1 Test TestL 14/11/12 AAAar/AAEAAAE5HAAB 2 Test TestL2 14/11/12 AAAar/AAEAAAE5HAAC 3 Test TestLC 14/11/12

Supplemental loggingSupplemental logging

New object

Existing

object

Supplemental logging is critical to healthy replication specially for tables with update/delete changes

KEYCOLs

Supplemental loggingSupplemental logging

New object

Existing

object

KEYCOLs

DDLOPTIONS &ADDTRANDATA &GETREPLICATES &

REPORT

To add SL to source DB

To replicat add SL to target

Supplemental LoggingTime

Source

t1 create table repuser2.tbl11 (id number, name varchar2(10));

2012-11-09 14:58:58 INFO OGG-01487 Oracle GoldenGate Capture for Oracle, ext_nb.prm: DDL found, operation

[create table repuser2.tbl11 (id number, name varchar2(10))2012-11-09 14:58:59 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

Source

t2 alter table repuser2.tbl11 add primary key (id);

[alter table repuser2.tbl11 add primary key (id) (2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

2012-11-09 15:00:04 INFO OGG-00479 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully deleted TRAN

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 105)].

2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGS_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

Target

2012-11-14 09:51:14 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggadmin].

No SLoperatio

Target

2012-11-14 09:51:14 INFO OGG-01408 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: Restoring current schema for DDL operation to [ggadmin].

no rows

No SLoperatio

Extract USER ID and Replicat USER ID should be different

• Problem documented in MOS 1472420.1“If your TRANSLOGOPTIONS EXCLUDEUSER specified in the Extract is the same as the EXTRACT USERID, the DDL to add supplemental logging is not captured and sent to the target.”

• Our extract settings were• USERID ggadmin

• TRANLOGOPTIONS EXCLUDEUSER ggadmin

Source

t2 alter table repuser2.tbl11 add primary key (id);

[alter table repuser2.tbl11 add primary key (id) (2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG DATA (ALL) COLUMNS /* GOLDENGATE_DDL_REPLICATION */ (size 105)].

2012-11-09 15:00:04 INFO OGG-00477 Oracle GoldenGate Capture for Oracle, ext_nb.prm: Successfully added TRAN

DATA for table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGS_61549" ("ID") ALWAYS /* GOLDENGATE_DDL_REPLICATION */ (size 117)].

DATA for [ALTER] DDL operation, table [REPUSER2.TBL11], operation [ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" /* GOLDENGATE_DDL_REPLICATION */ (size 104)].

Target

t1,t2 2012-11-09 14:59:01 INFO OGG-00489 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL is of mapped scope,

after mapping new operation [create table REPUSER2."TBL11" (id number, name varchar2(10)) (size 62)].

2012-11-09 15:00:06 INFO OGG-00482 Oracle GoldenGate Delivery for Oracle, rena_nb.prm: DDL found, operation

[alter table repuser2.tbl11 add primary key (id) (size 49)].

[ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS (size 72)].

[ALTER TABLE "REPUSER2"."TBL11" ADD SUPPLEMENTAL LOG GROUP "GGT_61549" ("ID") ALWAYS (size 85)].

[ALTER TABLE "REPUSER2"."TBL11" DROP SUPPLEMENTAL LOG GROUP "GGT_61549" (size 72)].

GETREPLICATES does not work as it is expected

Source

t1 Create table

t2 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS

t3 Create PK index

t4 add supplemental log data (id) (temp)

t5 drop SUPPLEMENTAL LOG DATA (ALL)

T6 add supplemental log data (id)

T7 drop supplemental log data (id) (temp)

Target

t1 Create table

t2 Create PK index

t3 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS

t4 add supplemental log data (id) (temp)

t5 drop supplemental log data (id) (temp)

1. ADDTRANDATA on source for new object.

2. Monitor replicated object on target and add SL manually before any DML change

SCHEMATRANDATAIn GoldenGate 11.1

OGG-01783 Cannot verify existence of table function that is required to enable schema

level supplemental logging, failed to find function

DB Patches needed to support SCHEMATRANDATA in OGG (Oracle GoldenGate), OGG-01783 Cannot verify existence

of table function that is required to enable schema [MOS 1426440.1]

To use SCHEMATRANDATA, please apply the fix in bug 13794550 Only on some

platforms

• Supplemental logging is critical for proper function

• Different users needed for extract and replicat

• Add trandata manually for replicated nodes

• Need to monitor schema for tables without trandata

• Use SCHEMATRANDATA if you can

Lessons Learned

DDL ReplicationTime Source Target

T1 create table luke ( a number ,b date ,c varchar2(30) );

alter table luke add primary key ( a );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30)

add trandata luke

T2 insert into luke values ( 1, sysdate, 'Luke Test 1' );

add trandata luke

T2 insert into luke values ( 1, sysdate, 'Luke Test 1' );

select * from luke;

A B C- --------- -------------------1 19-NOV-12 Luke Test 1

insert into luke values ( 2, sysdate, 'Luke Test 2' );

T3 select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1

update luke set c = 'Luke Test 1 updated' where a = 1;

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 21 19-NOV-12 Luke Test 1 updated

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 2 updated1 19-NOV-12 Luke Test 1 updated

alter table luke add ( d timestamp );

select * from luke;

A B C- --------- -------------------2 19-NOV-12 Luke Test 2 updated1 19-NOV-12 Luke Test 1 updated

alter table luke add ( d timestamp );

desc luke Name Null? Type ---- ----- ---- A NUMBER B DATE C VARCHAR2(30) D TIMESTAMP(6)

T5 insert into luke values ( 3, sysdate, 'Luke Test 3', systimestamp );

select a,d from luke;

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

insert into luke values ( 4, sysdate, 'Luke Test 4', systimestamp );

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

T6 select a,d from luke;

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 1 Missing

Value!

A D - ---------------------------- 2 3 22-NOV-12 09.40.30.622017 AM1

T6 select a,d from luke;

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 <<<*** MISSING ***>>>1

Stop extractStart extract

Discard -> Process Abending : 2012-11-22 09:53:07Source Context : SourceModule : [ggstd.conv.endian] SourceID : [/scratch/pradshar/view_storage/pradshar_bugdbrh40_12927937/oggcore/OpenSys/src/gglib/ggstd/lecnv.c] SourceFunction : [convCompSQL] SourceLine : [530] ThreadBacktrace : [9] elements : [/ggs/replicat(CMessageContext::AddThreadContext()+0x26) [0x5f1e56]] : [/ggs/replicat(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x7b2) [0x5e88f2]] : [/ggs/replicat(_MSG_ERR_MAP_COL_INDEX_INVALID(CSourceContext*, DBString<777> const&, int, int, CMessageFactory::MessageDisposition)+0xa6) [0x5bee16]] : [/ggs/replicat [0x900e6c]] : [/ggs/replicat(ggConvRecLE(char*, file_def*, int, char, char)+0x6c) [0x900fac]] : [/ggs/replicat [0x8f4d6e]] : [/ggs/replicat(main+0x84b) [0x507a0b]] : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x34cd81d994]] : [/ggs/replicat(__gxx_personality_v0+0x1e2) [0x4e3ffa]]

2012-11-22 09:53:07 ERROR OGG-01161 Bad column index (3) specified for table FEXLINK.LUKE, max columns = 3.

T7 Restart replicat

A D - ---------------------------- 3 22-NOV-12 09.40.30.622017 AM2 4 5 22-NOV-12 09.53.02.177904 AM1

• GoldenGate parameters (Replicat only)• DDLOPTIONS UPDATEMETADATA

• Automatically updates GoldenGate metadata cache

DDL Replication• UPDATEMETADATA doesn’t work both ways

1. After DDL ALTER must bounce REPLICAT on the local node and PRIMARY EXTRACT on other nodes

Conflict Avoidance• 2 Columns

• LAST_UPDATED_AT (DATE)

• LAST_UPDATING_BY (VARCHAR2(30))

• TriggerCREATE OR REPLACE TRIGGER <Owner>.<Table Name>_BIU BEFORE INSERT OR UPDATE ON <Owner>.<Table Name> FOR EACH ROW WHEN ( user != 'GGADMIN' ) DECLARE l_username VARCHAR2(50); BEGIN SELECT NVL(SYS_CONTEXT('USERENV','CLIENT_IDENTIFIER'),USER) INTO l_username FROM DUAL; :NEW.last_updated_at := SYSDATE; :NEW.last_updating_user := l_username; END; /

Conflict Avoidance

Sequences• 16 digit sequence number• Each silo has a unique 2 digit starting number e.g. 21, 31 etc• Ability to find originating silo for management reports• Guarantee uniqueness• Extra maintenance for new sequences• Need GG parameter

• DDL EXCLUDE OBJTYPE SEQUENCE

Conflict Detection

Discards• Check discard files

• Use a cron script or other scheduler for every 5 mins

• Discard reader built in PL/SQL• PIPELINE function• Parses the discard file for common errors• Condenses the output to one line with the main error information• Check blog for source code

• http://www.pythian.com/news/37703/read-goldengate-discard-files/

Conflict Detection

Discards• Usage of READ_DISCARD

• The discard had 567 lines• READ_DISCARD reduced output to 6 lines

select LINE_NUMBER , ORACLE_ERROR , SOURCE_OBJECT_NAME , ERROR_OPERATION , ERROR_COLUMN , ERROR_VALUE , PK_TABLE_NAMEfrom table(read_discard(‘repname’));

LINE_NUMBER ORA_ERROR SOURCE_OBJECT_NAME ERROR_OPERATION ERROR_COLUMN ERROR_VALUE PK_TABLE_NAME----------- --------- ------------------ --------------- ----------------- ---------------- ------------- 1 ORA-02291 CHILD_TABLE1 INSERT PARENT_ID 4100000000360956 PARENT 44 ORA-02291 CHILD_TABLE2 INSERT CHILD1_ID 6100000000041100 CHILD_TABLE1 63 ORA-02291 CHILD_TABLE3 INSERT PARENT_ID 4100000000360956 PARENT 263 ORA-01403 TABLE1 UPDATE TABLE1_ID 4100000000562453 311 ORA-01403 PARENT UPDATE PARENT_ID 4100000000360956 392 ORA-01403 CHILD_TABLE3 UPDATE CHILD3_ID 6100000000264611

6 rows selected.

Release Management

Major Release scriptMajor Release script

App code

DBchang

Pre-DDL

Post-DDL

DDLDDLPost-

Post-DDL

Pre-DDL

Statements which can not be replicated OR

We do not want to be replicatedRun these statements outside

replication cycleCREATE SEQUENCE

CREATE/ALTER MATERIALIZED VIEWCREATE SYNONYM on other user

Due to GG limitations

DMLDML

Statements which are replicated

via GoldenGate

Release Management

ERROR OGG-01161 Oracle GoldenGate Delivery for Oracle, resb01sa.prm: Bad column index (nn) specified for table mytab, max columns = 59.

ERROR OGG-01668 Oracle GoldenGate Delivery for Oracle, resb01sa.prm: PROCESS ABENDING.

Challenge – Replication from superset structure to minor set

Version N

Version N+1

Release Management

T Source Target

desc mytab

ID not null numberName varchar2(10)

desc mytab

Drop Column

Release Management

T Source Target

desc mytab

alter table mytab drop column name;

Drop Column

Release Management

T Source Target

desc mytab

alter table mytab drop column name; Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

insert into mytab values(1,’Test01’);

Drop Column

Release Management

T Source Target

desc mytab

-- ASSUMETARGETDEFSSOURCEDEFS ./dirsql/targetdefgen.sql

MAP mytab, TARGET mytab,COLMAP (ID=ID);

start replicat

Drop Column

Release Management

T Source Target

desc mytab

start replicat

start pump extract

Drop Column

Release Management

T Source Target

desc mytab

start replicat

start pump extract

select * from mytab

ID--------1

Select * from mytab

ID NAME---- -----------1 Test01

Drop Column

Release Management

T Source Target

desc mytab

Modify Column

Release Management

T Source Target

desc mytab

alter table mytab modify column name (varchar2(6));

Modify Column

Release Management

T Source Target

desc mytab

Generate defgen and copy to source./defgen paramfile ./dirprm/defgen.prm

Modify Column

Release Management

T Source Target

desc mytab

MAP mytab, TARGET mytab,COLMAP (USEDEFAULTS,NAME=@STREXT(NAME,1,6));

start replicat

Modify Column

Release Management

T Source Target

desc mytab

start replicat

start pump extract

Modify Column

Release Management

T Source Target

desc mytab

start replicat

start pump extract

Select * from mytab

ID NAME---- -----------1 Test01

Select * from mytab

ID NAME---- -----------1 Test011

Modify Column

Release Management

ASSUMETARGETDEFSASSUMETARGETDEFS

Old Version: Create metadata for New Version DB ( defgen )New Version: Use SOURCEDEFS & MAP for pulling data from superset structure

SOURCEDEFS & MAPSOURCEDEFS & MAP

Data ComparisonUser developed scripts• Master shell scripts calls PL/SQL package• Compares VARCHAR2, NUMBER, DATE and TIMESTAMP

• Large VARCHAR2, BLOB, CLOB fields are hashed for comparison

• Requires a database link• Uses LAST_UPDATED_AT timestamps to compare a subset of

each table e.g. SYSDATE – 7• Saves the ID of the offending row in a custom table• Inconsistencies are handled manually• Can be time consuming

Data ComparisonVeridata• Need to set up a server with a database component• Each database in the replication cluster needs an agent• There can be significant initial configuration for tables to

excluded columns, if necessary.• Hashes the entire row for less network traffic• All comparison is done on the veridata server• Reports are good and clear• Need to write an XML parser to create the insert / update /

delete statements• May need some clean up operations for report files on the

server

Trail file deletion• PURGEOLDEXTRACTS in mgr.prm

• PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 6

• Problems can occur if trail files have the same name (MOS 1460097.1)

• Best practice use unique trail file names (MOS 1324890.1)

• Use a script to check primary extracts, datapumps and replicats

• Script links Primary Extract -> DataPump(s) -> Replicat• Only delete once have been used by the next level

• Used by Replicat if you want to be really safe

• Parsing output from -> info <ER Name>, showch

Use unique trail file names

Trail file deletionInfo extract, showch output

Read Checkpoint #1

GGS Log Trail

Startup Checkpoint (starting position in the data source): Sequence #: 8 RBA: 1139 Timestamp: 2012-11-08 18:57:11.595452 Extract Trail: ./dirdat/wb

Current Checkpoint (position of last record read in the data source): Sequence #: 11 RBA: 1139 Timestamp: 2012-11-12 11:51:02.933110 Extract Trail: ./dirdat/wb

Write Checkpoint #1

GGS Log Trail

Current Checkpoint (current write position): Sequence #: 9 RBA: 5099 Timestamp: 2012-11-12 12:17:18.995401 Extract Trail: ./dirdat/wb

Archive Log Deletion• GoldenGate only sends transaction on commit• Need to keep archive logs until transaction

completes• Either user a script for archive log deletion• Or since 11.1.1.1 can register the extract (EE only)

• REGISTER EXTRACT <Extract Name> LOGRETENTION• DBA_CAPTURE -> OGG2$_EXT01UPL88C29FC8

• Get • RMAN-08137: WARNING: archived log not deleted, needed for

standby or upstream capture process

• Beware of bugs! (MOS 1487374.1)

Lessons Learned• Supplemental Logging is critical - get it right!• Structural changes need to be managed carefully• Parts of the database structure needs special attention• Sequences, Materialized Views, Extra columns

• Need to set up monitoring• Discards• Row comparison• GG processes and function

• Trail files need to be cleaned up• Archive log cleanup must be integrated with GG

Thank you and Q&A

http://www.pythian.com/news/

http://www.facebook.com/pages/The-Pythian-Group/163902527671

@pythian

http://www.linkedin.com/company/pythian

1-877-PYTHIAN

sales@pythian.com

To contact us…

To follow us…

@pythianjobs

experiences and lessons learned 6-node active-active oracle goldengate luke davies

oracle goldengate delivery

oracle goldengate capture

oracle exadata

oracle applications

oracle acesace directors

info ogg

operation alter table

target tbl4

Documents

oracle goldengate a- · pdf fileoracle goldengate best...

oracle® goldengate

oracle goldengate on disaster recovery - doag.org · >...

oracle goldengate 12c · release notes for oracle...

oracle goldengate administering oracle goldengate for...

insync10 goldengate

oracle goldengate faq...oracle goldengate faq

goldengate 12c

oracle goldengate on linux for...

oracle® goldengate application adapters for base24 · pdf...

oracle goldengate 11g overview - gcoug goldengate 11g...

oracle goldengate installing and configuring oracle...

goldengate catalog 2014

oracle maximum availability architecture (maa) goldengate...

oracle goldengate monitoring

oracle goldengate director

oracle goldengate faq

goldengate - replication

coherence goldengate hotcache

oracle goldengate windows and unix administrator’s … ·...