BRSPACE Online ReorganizationsTop Ten Pitfalls and Problems

Martin Frauendorfer, SAP Active Global Support

martin.frauendorfer@sap.com

November 2009

© SAP 2009 / Page 2

1. Overview2. Pitfalls and Problems

Agenda

© SAP 2009 / Page 3

Overview (1)

Reorganizations in general

A reorganization in a database context is the recreation orcleanup of one or several segments.Several types of reorganization exist:

Index reorganization (e.g. via REBUILD or COALESCE), mainly used toreduce index fragmentationTable reorganization with tools like:

– BRSPACE (online, offline, export / import)– Oracle tools (DBMS_REDEFINITION, MOVE, EXP / IMP, EXPDP /

IMPDP, …)– SAP functionality (ICNV, SE14, …)

Tablespace reorganization (reorganization of all tables in the tablespaceand recreation of the tablespace)

In this presentation we concentrate on the BRSPACE onlinereorganization of tables (which also includes tablespacereorganizations).

© SAP 2009 / Page 4

Overview (2)

In which situations can a table reorganization be useful?

Reduction of table sizeCleanup of table fragmentationChange of table storage parameters (e.g. INITRANS)Move of table to a different tablespaceReduction of chained and migrated rowsReduction of number of allocated extentsTransition from LONG to LOB columns (Oracle >= 10g)Reduction of hot spots on disk levelActivation of Transparent Data EncryptionChange of table structure

© SAP 2009 / Page 5

Overview (3)

In which situations can a tablespace reorganization be useful?

Activation of LMTSActivation of ASSMCleanup of freespace block corruptionReduction of database sizeReduction of backup sizeReduction of filesystem sizeReduction of fragmentationReduction of number of datafilesIncrease of number of datafilesTransition to new tablespace layoutOther change to tablespace layoutOptimization of filesystem layout

© SAP 2009 / Page 6

Overview (4)

How does a BRSPACE online reorganization work?Determination of CREATE commands for target table and indexes basedon DBMS_METADATA.GET_DDLCreation of target table with naming convention <source_table>#$Export of source table CBO statistics based onDBMS_STATS.EXPORT_TABLE_STATSCopy of source table data to target table based onDBMS_REDEFINITION.START_REDEF_TABLE. This package will implicitlycreate a materialized view log MLOG$_<source_table> to track allchanges that are performed on the source table while the reorganizationis running.Creation of target indexes with naming convention <source_index>#$Import of source table CBO statistics to target table based onDBMS_STATS.IMPORT_TABLE_STATSFinalizing of the online reorganization usingDBMS_REDEFINITION.FINISH_REDEF_TABLE. In this step all changes inthe materialized log are applied to the target table and the names ofsource and target table are exchanged.The source table is dropped.The „#$“ suffix of the target table indexes is removed.

© SAP 2009 / Page 7

Overview (5)

What are the advantages of a BRSPACE Online Reorganization?

It uses the advantages of DBMS_REDEFINITION:The reorganization can be performed in parallel to production operation.This means the table is accessible and records can be inserted / updated /deleted.If anything goes wrong during the reorganization the source table stillexists unchanged and no restore / recovery is necessary.Compared to an offline reorganization via export / import all data has to becopied only once and not twice

It provides possibilities for client and server side parallelism inorder to speed up the reorganization if sufficient systemresources exist and the reorganization should finish within aspecific time window.It has already proven to be a reliable reorganization tool at manycustomers.It performs a lot of useful activities implicitly that would require alot of manual efforts and impose the risk of errors ifDBMS_REDEFINITION is used directly.

© SAP 2009 / Page 8

Overview (6)

What are the restrictions of a BRSPACE Online Reorganization?

It is only available as of Oracle 9i.It cannot handle tables with LONG or LONG RAW columns. As of10g it is possible to convert them during an online reorganizationon the fly to LOB columns (if SAP >= 6.40 is used).Temporarily twice the space is needed because both source andtarget table and indexes exist in parallel.If a table has no primary key constraint (or no unique index withNOT NULL columns) a ROWID based online reorganization isnecessary that requires overhead (ROWID index, ROWID column).The setup of the materialized view is an overhead that cannegatively impact the performance of the reorganization of manyvery small tables.Structure changes on the source table and its indexes must notbe performed during the reorganization (as they would be lost).

© SAP 2009 / Page 9

Overview (7)

How is a normal BRSPACE Online Reorganization started?

Table reorganization:brspace -f tbreorg -t <table_name>

Tablespace reorganization:brspace -f tbreorg -s <source_tsp> -t "*" -n <target_tsp>

Additional useful options exist, e.g.:-p <degree>

BRSPACE parallelism degree-e <degree>

PX parallelism degree-l <category>

Initial extent size category-i <target_ind_tsp>

Definition of target index tablespace (if different from target tabletablespace)

For more detailed information see note 646681 and the BRSPACEonline documentation.

© SAP 2009 / Page 10

Pitfalls and Problems - Overview

Top Pitfalls and Problems1. What is a good parameter and redo log configuration?

2. What has to be considered for PSAPTEMP?

3. What about PSAPUNDO?

4. And why is the DEFAULT tablespace sometimes used significantly?

5. Why do tables sometimes grow during reorganization?

6. What about SAP technical settings?

7. Is it useful to convert from LONG to LOB?

8. What is a good way of parallelism?

9. Why should I look for invalid objects?

10. Why does a DROP sometimes take ages?

© SAP 2009 / Page 11

Pitfalls and Problems - 1. Configuration

1. Database ConfigurationA good online reorganization performance needs a slightlydifferent database configuration than normal system operation.Oracle parameters:

DB_FILE_MULTIBLOCK_READ_COUNT >= 128Fast full table scan on the source table when creating the target table

PGA_AGGREGATE_TARGET as large as possibleFast index creations in PGA with minimum PSAPTEMP accesses

If Oracle parallel execution is used:– PARALLEL_EXECUTION_MESSAGE_SIZE >= 16384– PARALLEL_MAX_SERVERS >= 2 * BRSPACE parallelism degree *

Oracle parallelism degree– PARALLEL_ADAPTIVE_MULTI_USER = FALSE

Otherwise Oracle may reduce the desired parallelism degree– SESSIONS, PROCESSES: Sufficiently large to cope with the

increased PARALLEL_MAX_SERVERS valueDB_CACHE_SIZE >= 1 GB

Large buffer pool doesn't provide significant advantage

© SAP 2009 / Page 12

Pitfalls and Problems - 1. Configuration

1. Database Configuration (2)Redo Log Configuration

Usually all changes of an online reorganization are recorded in the redologs, so there can be a significant amount of additional redo logsgeneratedWith current BRSPACE patches you can use the option "-NBR" to createthe target indexes with NOLOGGING

No redo log information is created for the indexesTheoretically it is possible to switch off logging for the target table, too.

Almost no redo log information created at allDangerous, because in case of recovery across the reorganization time

the table data is lost!Only useful during a dedicated reorganization time window and with a

good backup after the reorganizationRedo logs should be created large enough to avoid log switches within lessthan one minute

Avoids unnecessary checkpoint and log switch loadARCHIVELOG mode can be disabled in case of a dedicated reorganizationtime window and a good backup after the reorganization.

© SAP 2009 / Page 13

Pitfalls and Problems - 2. PSAPTEMP

2. PSAPTEMPProblems with PSAPTEMP can result in a long reorganizationruntime or terminations.PSAPTEMP recommendations:

Define PSAPTEMP as LMTS/T tablespace (see SAP Note 659946) inorder to avoid delays due to space transactionsMake sure that PSAPTEMP is assigned as temporary tablespace to theSAP user and to SYS.Make sure that there are no I/O bottlenecks when accessing PSAPTEMP.PSAPTEMP must be large enough to hold the size of the largestreorganized index.If BRSPACE parallelism is used ("-p <degree>" option), PSAPTEMP mustbe large enough for the <degree> largest indexes that are reorganized inparallel.In case of a sorted reorganization ("-r <index>" option), PSAPTEMP mustbe large enough to hold the table data.Consider AUTOEXTEND or RESUMABLE to be on the save side

© SAP 2009 / Page 14

Pitfalls and Problems - 3. PSAPUNDO

3. PSAPUNDOPSAPUNDO is only used to a minor extent:

The data transfer into the target table works with APPEND, so no undo isgenerated.The creation of the indexes also doesn't create undo.Concurrent DML operations logged in the MLOG$ table and finally appliedto the target table generate undo.

PSAPUNDO recommendations:Use Automatic Undo Management (AUM, SAP Note 600141)Usually no increase of PSAPUNDO necessaryLarger PSAPUNDO may be required if reorganization is done in parallel tosignificant DML operations (in general not recommended).

© SAP 2009 / Page 15

Pitfalls and Problems - 4. Default Tablespace

4. Default TablespaceThe default tablespace is used in two situations:

The MLOG$ table with the recorded DML operations is created in thedefault tablespace.If a table has no primary index, a ROWID index (I_SNAP$...) is created inthe default tablespace.

Default tablespace recommendations:Make sure that a default tablespace with sufficient free space orAUTOEXTEND is assigned to both the SAP user and SYS.Use LMTS or make sure that NEXT and MAXEXTENTS is set sufficientlyhigh in order to avoid max extents errors.

© SAP 2009 / Page 16

Pitfalls and Problems - 5. Table Growth

5. Table GrowthNormally a reorganization will reduce the table size or at leastkeep it on the same level.Sometimes the size reduction is not as high as expected or thereis even a size increaseThe most typical reasons are:

The reorganized table has no primary index.– In this case a ROWID based reorganization rather than a PK

constraint reorganization is done and a ROWID column has to beadded to the target table. This column is hidden after thereorganization, but it still allocates space ( table increase up to 30%)

– Typically happens with SAP conversion tables (QCM tables) and BWfact tables

– Should be avoided whenever possible (e.g. cleanup of QCM tables,MOVE based reorganization of BW fact tables)

A large INITIAL_EXTENT value is used– E.g. because of a former SAPDBA reorg with Compress Extents– Use BRSPACE option "-l 2" to set the initial extent to 64K.

© SAP 2009 / Page 17

Pitfalls and Problems - 6. Technical Settings

6. Technical SettingsThe SAP technical settings contain a data class (TABART) thatmaps a table to a tablespace:

DBSTATC SPROT PSAPPROTDDD09L TAORA /

IAORA

Table

6. Technical SettingsThe SAP technical settings contain a data class (TABART) thatmaps a table to a tablespace:

Table TABART Tablespace

The TABART is used whenever a table or index is newly created(e.g. during a transport of a new table or new index or during asystem copy). If an existing table or index is modified, the currenttablespace is used regardless of the TABART.Whenever you move tables and indexes to different tablespaces,you should consider the TABART settings.Otherwise segments may be created in wrong tablespaces.BRSPACE is TABART aware.

© SAP 2009 / Page 18

Pitfalls and Problems - 6. Technical Settings

6. Technical Settings (2)Recommendations:

Use the "-l" option to define an appropriate existing or new TABARTwhenever creating a new tablespace using "brspace -f tscreate":

– "-l <tabart>": New tabart <tabart>– "-l all": All standard TABARTs– "-l <source_tsp>": TABART of tablespace <source_tsp>

Maintain TABARTs also in the other systems of the system landscape.TABART can point to different tablespaces in different systems.Follow SAP Notes 778784 and 777615 in order to avoid problems duringsystem copies.In BW TABARTs are also maintained in special BW tables like RSDCUBE,RSDODSO, RSTS, RSDS and RSDCHABAS. These entries also have tobe adapted. See SAP Note 771191 for more information.

© SAP 2009 / Page 19

Pitfalls and Problems - 7. LONG2LOB

7. LONG2LOB ConversionTables with LONG and LONG RAW columns can't be reorganizedonline - unless an implicit LONG2LOB conversion is performed(BRSPACE parameter "-a long2lob")In general it is useful to convert to LOB columns, but there aresome restrictions:

Oracle >= 10g, SAP >= 6.40ORA-00932 errors possible on SAP side (for bugfix see SAP Note1223716)ORA-00600 errors possible if PX is used (for bugfix see SAP Note1292525)High water mark enqueues possible with LOBs in ASSM (for bugfix seeSAP Note 1166242).Several LOB related DBSL problems in SAP 7.1x (for bugfixes see SAPNotes 1329293, 1269704, 1257979, 1226314 and 1171043)Small performance impact due to LOBs reported (see SAP Note 835552),can usually be neglected

© SAP 2009 / Page 20

Pitfalls and Problems - 8. Parallelism

8. Parallelism SettingsTwo possibilities of parallelism:

"-p <degree>": BRSPACE parallelism degree, reorganization of multipledifferent tables in parallel"-e <degree>": Oracle parallel execution (PX) degree, several Oracle PXslaves work on the reorganization of one table in parallel

Reorganization in parallel to production operationSystem resources (I/O, CPU) mustn't be exceeded.Time is less criticalUse no or a rather small parallelism setting (e.g. "-p 2 - e 2")

Reorganization in a dedicated time window with low or no parallelproduction operation

Use the system resources (I/O, CPU) as good as possible.Often I/O is the bottleneck and not CPU, but I/O capacities are often hardto determine.Starting point can be e.g. "-p 6 -e 4" for tablespace reorganizations or "-e8" for individual table reorganization

© SAP 2009 / Page 21

Pitfalls and Problems - 9. Invalid Objects

9. Invalid objectsDifferent kinds of invalid objects can result in errors andterminations during online reorganizations.Therefore the following should be checked in advance:

Make sure that no index or index partition of the tables to be reorganized isin status UNUSABLE. Perform a rebuild if required.Make sure that no invalid triggers exist on the tables to be reorganized(e.g. /BIC/05… BW triggers). Drop or validate these triggers.Make sure that no SYS objects exist in the Oracle DDIC with status =INVALID. Particularly important are packages like DBMS_REDEFINITION,DBMS_METADATA, DBMS_STATS and underlying objects.Make sure that the source tables are consistent (SAP Note 23345). As anexample the REBUILD ONLINE bug described in SAP Note 1413928 canresult in ORA-01452 („cannot CREATE UNIQUE INDEX; duplicate keysfound“) errors when creating the unique index on the target table and thewhole table reorganization is terminated.

© SAP 2009 / Page 22

Pitfalls and Problems - 10. DROP Performance

10. DROP PerformanceUsually it is a matter of seconds (or less) to drop a table or anindex.But: Under certain circumstances dropping the source table (orthe source tablespace) can take longer than the wholereorganization. In real life we have already seen situations wherea DROP of a single table took several days!The reason are typically space transactions in dictionarymanaged tablespaces - every formerly allocated extent needs tobe freed individually. Due to an inadequate Oracle index designon the DDIC tables FET$ and UET$ the cleanup has a squarecomplexity (4 times higher runtime for twice the number ofextents).Even worse: As space transactions require the unique spacetransaction (ST) enqueue, other extent operations in the systemmay be blocked for a long time.

© SAP 2009 / Page 23

Pitfalls and Problems - 10. DROP Performance

10. DROP Performance (2)Recommendations

Check for critical segments in dictionary managed tablespaces with morethan 5000 extents. The higher the number of extents, the more critical.More than 100.000 extents is definitly a big problem.If segments are returned, you have several options:

– Reduce the number of extents in smaller portions in advance (e.g. bycopying and truncating the table with REUSE SPACE and doingALTER TABLE … DEALLOCATE UNUSED KEEP <size> with <size>decreasing over time)

– Reorganize the critical tables individually during certain definedtimeframes.

– Test how long the DROP takes and provide a sufficient time windowfor the reorganization in production.

– In extreme situations it can even make sense to recreate the databasewith LMTS using R3load.

See SAP Note 745639 (section TYPE = ST) for more information.

© SAP 2009 / Page 24

Thank you!

© SAP 2009 / Page 25

Copyright 2008 SAP AGAll rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changedwithout prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as theirrespective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned andassociated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, orother items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitationshall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in thesematerials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durchSAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Servicessowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähntenNamen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zuInformationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderenVereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmtenGeschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht dieRichtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, wederausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zwecksowie für die Gewährleistung der Nichtverletzung geltenden Rechts.SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendungdieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen überInternetseiten Dritter ab.Alle Rechte vorbehalten.