reducing redo

1 © 2005 Julian Dyke

ReducingRedo

Julian Dyke

Independent Consultant

Web Version

juliandyke.com

2

© 2005 Julian Dykejuliandyke.com

Agenda

Introduction Tests

Indexes Number of columns processed SELECT FOR UPDATE Number of rows processed COMMIT Batch size Global temporary tables External tables

Conclusion

3


Redo Records

Redo Block Header

16 bytes

Redo Block 512 or

1024 bytes

Redo Block Body

496 bytes

Redo Record 1

Redo Record 2

Wastage

Redo Record 3

Wastage

Header

Body

Body

Header

Spare

Header

Body

Spare

STOP

4


Change Vectors

Redo Record

Header

Body

Header

Body

Header

Body

Header

Body

ChangeVectors

ChangeVector 3

ChangeVector 1

ChangeVector 2

STOP

5


Header

Change Vector

Header

Body

STOP

6


Example Examples in this presentation taken from Formula 1 database

Contains full details of all races from 1961 to 2004

Updated annually in November (end of season)

Currently 20 cars per race 19 races per season `Approximately 360 new rows per season

juliandyke.net

7


Schema

CLASSIFICATION

SEASON

GRANDPRIX

RACE TEAMDRIVER

COUNTRYCIRCUIT

ENGINE

CAR

8


Cars Each season has up to 18 races (19 in 2005)

Each race has up to 39 entrants (13 races in 1989)

Each car has driver, team and engine laps completed (may be zero) optional notes

Results are classified as follows

C Classified

DNF Did not finish

DNS Did not start

DNQ Did not qualify

DIS Disqualified

9


Points Points basically awarded to driver and team as follows

1st 2nd 3rd 4th 5th 6th 7th 8th

Pre 1991 9 6 4 3 2 1

1991-2002 10 6 4 3 2 1

2003 onwards 10 8 6 5 4 3 2 1

But . . . not always straightforward Half points awarded for incomplete races Split races (two half point races aggregated) Driver and / or team disqualifications e.g. Tyrrell in 1984 Up to 1980 only best scores counted for each half of

season e.g. Best 5 results from first 7 races and best 5 results from

last 7 races 1982-1990 only best 11 results counted for drivers 1961-1978 only first car to finish counted for each team

10


Input file - car.csv Comma separated file 16181 rows Fields are:

season_key race_key position driver_key team_key engine_key laps_completed classification_key notes (optional)

2004 17 17 ZBAU MIN FOR 41 DNF Spin2004 17 18 DCOU MCL MER 38 DNF Accident2004 17 19 RBAR FER FER 38 DNF Accident2004 17 20 MWEB JAG FOR 20 DNF Overheated2004 18 1 JMON WIL BMW 71 C2004 18 2 KRAI MCL MER 71 C2004 18 3 RBAR FER FER 71 C2004 18 4 FALO REN REN 71 C2004 18 5 RSCH WIL BMW 71 C2004 18 6 TSAT BAR HON 71 C2004 18 7 MSCH FER FER 71 C2004 18 8 FMAS SAU FER 71 C2004 18 9 GFIS SAU FER 71 C2004 18 10 JVIL REN REN 70 C2004 18 11 DCOU MCL MER 70 C2004 18 12 JTRU TOY TOY 70 C2004 18 13 RZON TOY TOY 70 C2004 18 14 CKLI JAG FOR 69 C2004 18 15 TGLO JOR FOR 69 C2004 18 16 ZBAU MIN FOR 67 C2004 18 17 GBRU MIN FOR 67 C2004 18 18 MWEB JAG FOR 23 DNF Accident2004 18 19 NHEI JOR FOR 15 DNF Clutch2004 18 20 JBUT BAR HON 3 DNF Engine

11


Input file - points.csv Comma separated file 16181 rows Fields are:

season_key race_key position driver_points team_point

2004 17 17 0 02004 17 18 0 02004 17 19 0 02004 17 20 0 02004 18 1 10 102004 18 2 8 82004 18 3 6 62004 18 4 5 52004 18 5 4 42004 18 6 3 32004 18 7 2 22004 18 8 1 12004 18 9 0 02004 18 10 0 02004 18 11 0 02004 18 12 0 02004 18 13 0 02004 18 14 0 02004 18 15 0 02004 18 16 0 02004 18 17 0 02004 18 18 0 02004 18 19 0 02004 18 20 0 0

12


CAR table CAR table and index definitions

CREATE TABLE car(

season_key NUMBER NOT NULL,race_key NUMBER NOT NULL,position NUMBER NOT NULL,driver_key VARCHAR2(4) NOT NULL,team_key VARCHAR2(3) NOT NULL,engine_key VARCHAR2(3) NOT NULL,laps_completed NUMBER NOT NULL,classification_key VARCHAR2(4) NOT NULL,notes VARCHAR2(100),driver_points NUMBER NOT NULL DEFAULT 0,team_points NUMBER NOT NULL DEFAULT 0

);

ALTER TABLE car ADD CONSTRAINT car_pkPRIMARY KEY (season_key,race_key,position);

CREATE INDEX car_driver ON car (season_key,driver_key,driver_points);

13


CAR CAR table relational integrity definitions

ALTER TABLE car ADD CONSTRAINT car_raceFOREIGN KEY (season_key,race_key)REFERENCES race (season_key,race_key);

ALTER TABLE car ADD CONSTRAINT car_driverFOREIGN KEY (driver_key)REFERENCES driver (driver_key);

ALTER TABLE car ADD CONSTRAINT car_teamFOREIGN KEY (team_key)REFERENCES team (team_key);

ALTER TABLE car ADD CONSTRAINT car_engineFOREIGN KEY (engine_key)REFERENCES engine (engine_key);

ALTER TABLE car ADD CONSTRAINT car_classificationFOREIGN KEY (classification_key)REFERENCES classification (classification_key);

14


For each line in car.csv{

read :season_key, :race_key, :position, :driver_key, :team_key,:engine_key, :laps_completed, :classification_key, :notes;

INSERT INTO car (season_key, race_key, position,driver_key, team_key, engine_key,laps_completed, classification_key, notes)

VALUES(:season_key,:race_key,:position,:driver_key, :team_key, :engine_key,:laps_completed, :classification_key, :notes)

COMMIT;}

Baseline - Insert

15


Baseline - Insert

Redo Generation for each Insert Statement

Header 5.2 Start Transaction

5.1 (11.1)Undo Undo insert row in CAR table INSERT

11.2Redo Insert row in CAR tableINSERT

Undo Undo insert row into CAR_PK index5.1 (10.22)INSERT

Redo 10.2 Insert row into CAR_PK indexINSERT

5.1 (10.22)Undo Undo insert row into CAR_DRIVER indexINSERT

Redo 10.2 Insert row into CAR_DRIVER indexINSERT

Commit 5.4 End TransactionCOMMIT

Oracle 9.2 and below

16


Insert Statement

Redo Generation for eachInsert Statement









Oracle 10.1 and above

17


Baseline - UpdateFor each line in points.csv{

read :season_key, :race_key, :position, :driver_points, :team_points;

SELECT driver_key, team_key, engine_key, laps_completed, classification_key, notes

INTO :driver_key, :team_key, :engine_key, :laps_completed, :classification_key, :notes

FROM carWHERE season_key = :season_keyAND race_key = :race_keyAND position = :positionFOR UPDATE;

UPDATE car SETdriver_key = :driver_key, team_key = :team_key,engine_key = :engine_key,laps_completed = :laps_completed,classification_key = :classification_key,notes = :notes, driver_points = :driver_points, team_points = :team_points

WHERE season_key = :season_keyAND race_key = :race_keyAND position = :position;

COMMIT;}

18


Baseline - Update

Redo Generation for eachUpdate Statement


5.1 (11.1)Undo Undo update row in CAR table UPDATE

11.5Redo Update row in CAR tableUPDATE

Undo Undo delete row from CAR_DRIVER index5.1 (10.22)UPDATE

Redo 10.4 Delete row from CAR_DRIVER indexUPDATE

5.1 (10.22)Undo Undo insert row into CAR_DRIVER indexUPDATE

Redo 10.2 Insert row into CAR_DRIVER indexUPDATE

5.1 (11.1)Undo Undo lock row in CAR tableSELECT FOR UPDATE

11.4Redo Lock row in CAR tableSELECT FOR UPDATE


Oracle 9.2 and below

19


Baseline - Update

Redo Generation











Oracle 10.1 and aboveRedo Generation for eachUpdate Statement

20


Baseline - Results Redo Generation in Bytes

Operation INSERT(car.csv)

UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Note Amount of redo generated by both INSERT and UPDATE

can be variable due to Undo segment management Recursive DDL statements e.g. extent allocation Block cleanouts

21


Test 1 Check for unused indexes

CAR_PK indexes columns SEASON_KEY RACE_KEY POSITION

supports primary key therefore mandatory

CAR_DRIVER indexes columns SEASON_KEY DRIVER_KEY DRIVER_POINTS

no longer required by current version of application

DROP INDEX car_driver;

22


Test 1 - Insert

Redo Generation for each Insert Statement









STOP

23


Test 1 - Update

Redo Generation












STOP

24


Test 1 - Results Redo Generation in Bytes


UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test 1 14687756 12467400 27155156

Conclusion Eliminating redundant index reduced

insert redo generation by 5761096 bytes update redo generation by 1942276 bytes

25


Test 2 In UPDATE statements

For tables undo and redo is generated for all columns in SET clause

For indexes undo and redo are only generated for index keys that have changed

Statements often update all columns to reduce parsing e.g.:



26


Test 2 - Update

Redo Generation








Slot = 23Col 3 = JMONCol 4 = WILCol 5 = BMWCol 6 = 71Col 7 = C Col 8 = <Null>Col 9 = 10Col 10= 10


11.5Redo

5.1 (11.1)Undo



STOP

27


Test 2 Only update columns which can have new values

DRIVER_POINTS TEAM_POINTS

For each line in points.csv{


SELECT ... FOR UPDATE;



COMMIT;}



SELECT ... FOR UPDATE;



COMMIT;}

28




UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test1 14687756 12467400 27155156

Test2 14560052 11584760 26144812

Conclusion Eliminating unnecessary columns from update statements

reduced update redo generation by 882640 bytes Would be significantly more if unchanged columns

included long fields e.g. CHAR, or VARCHAR2

29


Test 3 Eliminate unnecessary SELECT FOR UPDATE statements





FROM carWHERE season_key = :season_key AND race_key = :race_key AND position = :positionFOR UPDATE;

UPDATE car SET driver_points = :driver_points, team_points = :team_points

WHERE season_key = :season_keyAND race_key = :race_key AND position = :position;

COMMIT;}





FROM carWHERE season_key = :season_key AND race_key = :race_key AND position = :positionFOR UPDATE;



COMMIT;}

30


Test 3 - Update

Redo Generation








STOP

31




UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test 1 14687756 12467400 27155156

Test 2 14560052 11584760 26144812

Test 3 14554428 8475484 23029912

Conclusion Eliminating SELECT FOR UPDATE statement reduced

update redo generation by 3109276 bytes

32


Test 4 Rows are inserted with default values of 0 for driver_points

and team_points Points only scored by

first eight cars - 2003 onwards first six cars - pre 2003

Only update rows with non-zero rows for driver_points and/or team_points

Team

Driver

Points No Points

Points 3514 30

No Points 324 12313

Driver Team

324 3514 30 12313

STOP

33


Header 5.2

11.5Redo

5.1 (11.1)Undo

Commit 5.4

Test 4 - Update

Redo GenerationRedo Generation for eachUpdate Statement

Header 5.2

11.5Redo

5.1 (11.1)Undo

Commit 5.4

UPDATE car SET driver_points = 1 team_points = 1

WHERE ...

col9 = 0 col10 = 0

col9 = 1 col10 = 1


WHERE ...

col9 = 0 col10 = 0

col9 = 0 col10 = 0


WHERE ...

col9 = 0 col10 = 0

col9 = 0 col10 = 0


WHERE ...

col9 = 0 col10 = 0

col9 = 9 col10 = 9

Header 5.2

11.5Redo

5.1 (11.1)Undo

Commit 5.4

Header 5.2

11.5Redo

5.1 (11.1)Undo

Commit 5.4

Header 5.2

11.5Redo

5.1 (11.1)Undo

Commit 5.4


WHERE ...

col9 = 0 col10 = 0

col9 = 9 col10 = 9

STOP

34


Test 4 Only update rows with non-zero rows for driver_points and/or

team_points



UPDATE car SETdriver_points = :driver_points, team_points = :team_points


COMMIT;

}



IF driver_points != 0 OR team_points != 0 THEN {

UPDATE car SETdriver_points = :driver_points, team_points = :team_points


COMMIT;}

}

35




UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test 1 14687756 12467400 27155156

Test 2 14560052 11584760 26144812

Test 3 14554428 8475484 23029912

Test 4 14683408 2070316 16753724

Conclusions Eliminating unnecessary update statements reduced

update redo generation by 6405168 bytes

36






COMMIT;}





COMMIT;}

COMMIT;

Test 5 Eliminate unnecessary COMMIT statements

37







COMMIT;}

}






COMMIT;}

}

COMMIT;

Test 5 Eliminate unnecessary COMMIT statements (continued)

38


Test 5 - Insert

Redo GenerationRedo Generation for eachInsert Statement













STOP

39




UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test 1 14687756 12467400 27155156

Test 2 14560052 11584760 26144812

Test 3 14554428 8475484 23029912

Test 4 14683408 2070316 16753724

Test 5 9516512 1028084 10544596

Conclusion Eliminating COMMIT statements reduced

insert redo generation by 5166896 bytes update redo generation by 1042232 bytes

40


Test 6 Default batch size is 1 Test INSERT and UPDATE with different batch sizes

Batch Size INSERT Redo UPDATE Redo Total Redo

1 9517096 1028084 10545180

2 5654136 1028084 6682220

4 3927092 1028440 4955532

8 3011944 1028084 4040028

16 2588540 1028636 3617176

32 2375884 1028172 3404056

64 2254936 1028040 3282976

128 2195876 1028084 3223960

256 2179404 1028440 3207844

512 2163816 1028084 3191900

1024 2163084 1028084 3191168

2048 2160012 1028084 3188096

41


Test 6 - Results

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

9000000

10000000

1 2 4 8 16 32 64 128 256 512 1024 2048

Batch Size

Red

o (b

ytes

)

42




UPDATE(points.csv)

Total

Baseline 20448852 14409676 34858528

Test 1 14687756 12467400 27155156

Test 2 14560052 11584760 26144812

Test 3 14554428 8475484 23029912

Test 4 14683408 2070316 16753724

Test 5 9516512 1028084 10544596

Test 6 2195876 1028084 3223960

Conclusion Batch Size of 128

reduced insert redo generation by 7320636 bytes update redo generation unaffected

43


Test 7 Create global temporary table

CREATE GLOBAL TEMPORARY TABLE temporary_car (

season_key VARCHAR2(4),race_key VARCHAR2(2),position NUMBER,driver_key VARCHAR2(4),team_key VARCHAR2(3), engine_key VARCHAR2(3),laps_completed NUMBER,classification_key VARCHAR2(4),notes VARCHAR2(100),driver_points NUMBER,team_points NUMBER

)ON COMMIT PRESERVE ROWS;

44


Test 7 Insert rows into global temporary table



INSERT INTO temporary_car (season_key, race_key, position,driver_key, team_key, engine_key,laps_completed, classification_key, notes)


COMMIT;}

Generated 64140 bytes of redo

45


Test 7 Update points in global temporary table




UPDATE temporary car SET driver_points = :driver_points, team_points = :team_points


}}

COMMIT;


46


Test 7 Copy rows from global temporary table to permanent table

INSERT INTO car (

season_key, race_key, position, driver_key, team_key, engine_key, laps_completed, classification_key, notes, driver_points, team_points ) SELECT

season_key, race_key, position, driver_key, team_key, engine_key, laps_completed, classification_key, notes, driver_points, team_points FROM temporary_car;


APPEND hint had no effect

47



Conclusion Global Temporary Table reduced total redo generation by

340212 bytes

Operation Description Total

Baseline Update all rows 34858528

Test 1 Update affected rows 27155156

Test 2 Update affected columns 26144812

Test 3 Drop index 23029912

Test 4 SELECT FOR UPDATE 16753724

Test 5 COMMIT 10544596

Test 6 Increase Batch Size 3223960

Test 7 Global Temporary Table 2883748

48


Test 8 Create external tables

CREATE OR REPLACE DIRECTORY external_dir AS '/u01/app/oracle/gp';

CREATE TABLE external_points (

season_key VARCHAR2(4),race_key VARCHAR2(2),position NUMBER,driver_points NUMBER,team_points NUMBER

)ORGANIZATION EXTERNAL(

TYPE ORACLE_LOADERDEFAULT DIRECTORY external_dirACCESS PARAMETERS(

RECORDS DELIMITED BY NEWLINEFIELDS TERMINATED BY ','

)LOCATION ('points.csv')

);

49


Test 8CREATE TABLE external_car (

season_key VARCHAR2(4),race_key VARCHAR2(2),position NUMBER,driver_key VARCHAR2(4),team_key VARCHAR2(3), engine_key VARCHAR2(3),laps_completed NUMBER,classification_key VARCHAR2(4),notes VARCHAR2(100)

)ORGANIZATION EXTERNAL(

TYPE ORACLE_LOADERDEFAULT DIRECTORY external_dirACCESS PARAMETERS(

RECORDS DELIMITED BY NEWLINEFIELDS TERMINATED BY ','MISSING FIELD VALUES ARE NULL

)LOCATION ('car.csv')

);

50


Test 8 Insert directly into permanent table joining contents of both

external tables

INSERT INTO car (

season_key, race_key, position, driver_key, team_key, engine_key, laps_completed, classification_key, notes,

driver_points, team_points ) SELECT

c.season_key, c.race_key, c.position, c.driver_key, c.team_key, c.engine_key, c.laps_completed, c.classification_key, c.notes, p.driver_points, p.team_points

FROM external_car c, external_points p WHERE c.season_key = p.season_key AND c.race_key = p.race_keyAND c.position = p.position";


51



Conclusion External Tables reduced total redo generation by 717024

bytes

Operation Description Total

Baseline Update all rows 34858528

Test 1 Update affected rows 27155156

Test 2 Update affected columns 26144812

Test 3 Drop index 23029912

Test 4 SELECT FOR UPDATE 16753724

Test 5 COMMIT 10544596

Test 6 Increase Batch Size 3223960

Test 7 Global Temporary Table 2883748

Test 8 External Table 2166724

52


Conclusion We have seen that the following techniques can be used to

reduce the amount of redo generated:

Eliminating redundant indexes Reducing the number of columns updated Eliminating redundant SELECT FOR UPDATE statements Reducing the number of rows processed Eliminating COMMIT statements Increasing the batch size Using Global Temporary Tables Using External Tables

53


Thank you for your interest

For more information and to provide feedback

please contact me

My e-mail address is:[email protected]

My website address is:

www.juliandyke.com

reducing redo

Documents