Index Rebuild Performance

Hopefully you’ll never need it.

Wei QiuPrinciple EngineerProgress Software Inc

Agenda

1 Overview

Processing Details

Tuning Suggestions

Some Statistics

2

3

4

Summary5

Phases of Index Rebuild (“non-recoverable”)

Index Scan

Data Scan/Key Build

Sort-Merge

Index KeyInsertion

• Scan index data area start to finish

• I/O Bound with little CPU activity

• Eliminated with area truncate

• Scan table data area start to finish (area at a time)

• Read records, build keys, insert to temp sort buffer

• Sort full temp file buffer blocks (write if > -TF)

• I/O Bound with CPU Activity

• Sort-merge –TF and/or temp sort file

• CPU Bound with I/O Activity

• I/O eliminated if –TF large enough

• Read sorted list in –TF or temp sort file

• Insert keys into index

• Formats new clusters; May raise HWM

• I/O Bound with little CPU Activity

Index Rebuild Parameters - Overview

sort block size (8K – 64K, note new limit)

# threads for data scan phase

merge block size ( default -TB)merge pool fraction of system memory (in %)

# threads per concurrent sort group merging

-mergethreads

# concurrent sort group merging-threadnum# merge buffers to merge each merge pass

-TM

report system usage statistics

a bit quieter than before

-rusage

-silent

-TB-

datascanthreads

-TMB

-TF

Agenda

1

Processing Details

Overview

Tuning Suggestions

Some Statistics

2

3

4

Summary5

Phases of Index Rebuild

Index Scan• Scan index data area start to

finish• I/O Bound with little CPU activity• Eliminated with area truncate

Area 9: Index scan (Type II) complete.

• Index area is scanned start to finish (single threaded)• Block at a time with cluster hops

• Index blocks are put on free chain for the index• Index Object is not deleted (to fix corrupt cluster or block

chains)• Order of operation:

• Blocks are read from disk,• Blocks are re-formatted in memory• Blocks are written to disk as –B is exhausted

• Causes I/O in other phases for block re-format• Can be eliminated with manual area truncate where possible

Phases of Index Rebuild

Data Scan/Key Build

• Scan table data area start to finish (area at a time)

• Read records, build keys, insert to temp sort buffer

• Sort full temp file buffer blocks (write if > -TF) • I/O Bound with CPU Activity

Index Scan• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate

• Table data area is scanned start to finish (multi-threaded if –datascanthreads)• Each thread processes next block in area (with cluster hops)

• Database re-opened by each thread in R/O mode• Ensure file handle ulimits set high enough

Processing area 8 : (11463) Start 4 threads for the area. (14536)Area 8: Multi-threaded record scan (Type II) complete.

Rules for -datascanthreads

Index rebuild run “with sort” option No index being rebuilt exists in table area

being scanned No word indexes are being rebuilt for table

data being scanned Data area being scanned is type TII storage

area.

If any true:◦ use single threaded data scan for that area◦ Violation for one area does not preclude use

on another area

Data Scan/Key Build

Record b) Extract next record from data block and build index keys (sort order)Key

-TF

c) Insert key into sort block for sort group(-TB 8K thru 64K)

d) Sort/merge full sort block into merge block.(-TMB -TB thru 64K)

Sort Block

a) Thread reads next data block in data area

RM Block

e) Write merge block to –TF, overflow to temp(-TMB sized I/O)

.srt1 .srt2

DB

…

Merge Block

What about “mixed” areas?Process index blocks as in index scan phaseNo longer read only!

Sort Block

Sort Groups: -SG 8 (note 8 is minimum)

Each sort group has its own sort file Sort file location

◦ 1 & 2. Sort files in same directory(I/O contention)

◦ 4. Sort files in different location Ensure enough space

Record

Index 1SG 1

SG 2

SG 3

Index 2

Index 3

Index 9

1) -T /usr1/richb/temp/

4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/

.srt1

.srt2

.srt3

2) <dbname>.srt 0 /usr1/richb/temp/

Each index assigned a particular sort group (hashed index #)

3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/

More on Sort Groups

Sort file size is in 1K units◦ 0 indicates unlimited space

Sort file max size with –TMB 8: 16 TB◦ Increase –TMB (-TMB 64: 128 TB)

3) <dbname>.srt 10240 /usr1/richb/temp/ 0 /usr1/richb/temp/

Record

Index 1SG 1

SG 2

SG 3

Index 2

Index 3

Index 9

1) -T /usr1/richb/temp/

4) <dbname>.srt 0 /usr1/richb/temp/ 0 /usr2/richb/temp/ 0 /usr3/richb/temp/

.srt1

.srt2

.srt3

2) <dbname>.srt 0 /usr1/richb/temp/

Each index assigned a particular sort group (hashed index #)

Last one on Sort Groups, I promise

Max –SG 64 per area. What if more than 64 indexes◦ Sort groups can contain more than one index’s entries: MOD(idx-

num, -SG)◦ Data scan will add key entry to appropriate sort group◦ Sort/merge orders indexes in sort group by index number◦ Key insertion phase inserts all entries for one index followed by all

index key entries for the other within same sort group.

Record

Index 1

SG 1

SG 2

SG 3

Index 2

Index 3

Index 9 .srt1

.srt2

.srt3

Each index assigned a particular sort group (hashed index #)

Index 1 Index 9

Index 2

Index 3

Phases of Index Rebuild

Index Scan

Data Scan/Key Build

• Sort-merge –TF and/or temp sort file

• CPU Bound with I/O Activity• I/O eliminated if –TF large enough

Sort-Merge

• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate

• Scan table data area start to finish (area at a time)

• Read records, build keys, insert to temp sort buffer

• Sort full temp file buffer blocks (write if > -TF)

• I/O Bound with CPU Activity

Sorting index group 3Spawning 4 threads for merging of group 3.Sorting index group 3 complete.

Sort-Merge Phase

Sort blocks in each sort group have been sorted and merged into a linked list of individual merge blocks◦ stored in –TF and temp files.

Merge blocks are further merged◦ –TM# at a time to form new larger “runs” of sorted merge

blocks. -TM# of these new “runs” are then merged

◦ forming even larger “runs” of sorted merge blocks. When only one very large “run” left, all the key entries in the

sort group are in sorted order.

Sorted!

Up to 7 concurrent merge threads

Up to 3 concurrent merge threads

Only 1 merge thread on last run

-threadnum vs -mergethreads-threadnum 2

-TF.srt1

-TF.srt2

-TF.srt3

Thread 1

Thread 2 Merge phase group 2

Merge phase group 1

-threadnum vs -mergethreads -threadnum 2

-TF.srt1

-TF.srt2

-TF.srt3

Thread 2

Thread 1

Merge phase group 2

Merge phase group 3

Thread 0

B-tree insertion occurs as soon as a sort group’s merge is completed.

Thread 0 begins b-tree insertionconcurrently.

-threadnum vs -mergethreads -threadnum 2 –mergethreads 3

Thread 1

Thread 2

Note: 8 actively running threads

Thread 3

Thread 4

Thread 5

Thread 6

Thread 7

Thread 8

-TF.srt1

-TF.srt2

-TF.srt3

Merge phase group 1

Merge phase group 2

Merge threads merge successive “runs” of merge blocks concurrently.

-threadnum vs -mergethreads -threadnum 2 –mergethreads 3

Thread 2Thread 6

Thread 7

Thread 8

Thread 1Thread 3

Thread 4

Thread 5

-TF.srt2

-TF.srt3

-TF.srt1

Merge phase group 3

Merge phase group 2

-threadnum vs -mergethreads -threadnum 2 –mergethreads 3

Thread 0

Thread 2Thread 6

Thread 7

Thread 8

Thread 1Thread 3

Thread 4

Thread 5

-TF.srt1

-TF.srt2

-TF.srt3

Thread 0 begins b-tree insertionconcurrently.

Best performance withlow –threadnum & high -mergethreads

Merge phase group 2

Merge phase group 3

Note: 9 actively running threads

B-tree insertion occurs as soon as a sort group’s merge is completed.

Phases of Index Rebuild

Index Scan

Data Scan/Key Build

Sort-Merge

• Scan index data area start to finish• I/O Bound with little CPU activity• Eliminated with area truncate

• Scan table data area start to finish (area at a time)

• Read records, build keys, insert to temp sort buffer

• Sort full temp file buffer blocks (write if > -TF)

• I/O Bound with CPU Activity• Sort-merge –TF and/or temp sort

file• CPU Bound with I/O Activity• I/O eliminated if –TF large enough

Index KeyInsertion

• Read sorted list in –TF or temp sort file

• Insert keys into index• Formats new clusters; May raise

HWM• I/O Bound with little CPU Activity

Index Key Insertion Phase

Key entries from sorted merge blocks are inserted into b-tree

Performed sequentially entry at a time, index at a time Leaf level insertion optimization (avoids b-tree scan) Leaf level written to disk as soon as full (since never

revisited)

Building index 11 (cust-num) of group 3 …Building of indexes in group 3 completed.Multi-threaded index sorting and building complete.

Root

Leaf Leaf Leaf

Write leaf when full

DB

Index B-tree

2085 Indexes were rebuilt. (11465)Index rebuild complete. 0 error(s) encountered.

Agenda

1

Tuning Suggestions

Overview

Some Statistics

2

3

4

Summary5

Processing Details

Assumptions for best performance

Index data is segregated from table data◦ Indexes & tables are in different storage areas◦ User data areas are TII storage areas

You have enough memory/disk space for sorting◦ If not, go home. You’re done.

You understand the impact of CPU and memory consumption◦ Process allowed to use available system

resources

Index Rebuild - Tuning

Truncate “index only” area if possible

.srt file – set up properly◦ Spread I/O across disks◦ Avoid with –TF settings (or RAM disk)

The contents of table “customer" will be deleted.

The contents of index "cust-order" will be deleted.

Are you sure you want to truncate storage area "Customer/Order area" (y/n)

proutil <db> -C truncate area “Customer/Order Area”

This could be a life changing decision…

Index Rebuild - TuningParameters

◦ -datascanthreads = 1.5 * # CPUs◦ -mergethreads * -threadnum = 1.5 * #CPUs◦ -threadnum 2 to 4◦ -B 1024◦ –TF 80

monitor physical memory paging◦ –TMB 64◦ –TB 64◦ –TM 32

could be lower with large –mergethreads (> 16)

Memory usage approximation

Data Scan overhead:

dataScanMemory = -datascanthreads * (-TMB + 3,072)

eachSortGroup = ((-TMB * -TM) + TMB) * -mergethreads

Memory allocated for all sort groups:

allSortGroups = eachSortGroup * active sort groups

 -TF memory merge pool fraction:

tfMemory = MIN((physicalMemory - InuseMemory),

((physicalMemory * -TF) / 100))

 Total memory used (in KB):

totalMemory = dataScanOverhead + tfMemory + allSortGroups + 1MB

A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes

proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8

Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632

A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes

proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8

Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632

Memory allocated for all sort groups:allSortGroups = eachSortGroup * 10 = 166,320

A“rough” approximation: 64 GB Ram, 16 CPUs, 8 indexes

proutil <db> -C idxbuild area “Customer/Order Index Area” -B 1024 -SG 10 -TB 64 -TM 32 -TMB 64 –TF 80 -datasanthreads 24 -threadnum 3 -mergethreads 8

Data Scan overhead:dataScanMemory = 24 * (64 + 3,072) = 76,800eachSortGroup = ((64 * 32) + 64) * 8 = 16,632

Memory allocated for all sort groups:allSortGroups = 16,632 * 10 = 166,320

 -TF memory merge pool fraction: tfMemory = MIN((64GB - InuseMemory), ((64GB * 80) /

100))

 Total memory used (in KB):totalMemory = 76,800+ 53,687,091 + 166,320 + 1MB = 54,909,667 K = 52.4 GB

Agenda

1

Some Statistics

Overview

Tuning Suggestions

Processing Details2

3

4

Summary5

Performance Numbers

0

20,000

40,000

60,000

80,000

100,000

120,000

Index Rebuild

10.2b06 best

10.2b06 no truncate

10.2b06 w/-TF 50

10.2b06 baseline

Elapsed Time

Cost of eachphase (in secs)

12 ½ hours 2 ½ hours

5X improvement!

Agenda

1

Summary

Overview

Tuning Suggestions

Some Statistics

2

3

4

Processing Details

5

Summary

Index Rebuild◦ Big improvements if

Your database is setup properly You provide system resources to index rebuild You use the new settings in 10.2b06

◦ One bug fix in 10.2b07◦ More efficient memory allocation coming soon!

(10.2b08)◦ Hopefully you’ll never need to index rebuild

Questions?

October 6–9, 2013 • Boston #PRGS

www.progress.com/exchange-pug

Special low rate of $495 for PUG Challenge attendees with the code PUGAM

And visit the Progress booth to learn more about the Progress App Dev Challenge!