Improving Maintenance and Reducing Patching Downtime in 11i

Ric Ginsberg

Vice PresidentApplications Release EngineeringOracle Corporation

Topics

Patching Best Practices

Enhancements In Release 11i

Maintenance Initiatives

Best Practices (#1)

Schedule periodic downtime for proactive maintenance

Keep up-to-date on latest AD code

Keep your test system current with production

Perform “uptime” maintenance– Gather schema stats

– Patch help system

– Apply translation database updates

Best Practices (#2)

Use a shared APPL_TOP and Distributed AD

Avoid extent allocation errors with autoextend

Combine patches with AD Merge Patch Use AD Patch ‘novalidate’ option

(default as of AD.H)

Use a staged APPL_TOP

Enhancements InRelease 11i

11i Installation Enhancements

Eliminates error prone OUI/ORCA based install

Uses Cloning architecture

Installs in about an hour

Installs translation filesystem (11.5.9)

Rapid Install “-techstack” option

– iAS 1.0.2.2.2

11i Performance Enhancements (#1)

Faster driver read (50x faster)

Faster prereq checking

Faster file copy

Faster Patch History processing

Reduced worker idle time

Increased worker parallelism

11i Performance Enhancements (#2)

Create PL/SQL packages directly

Generate reports in parallel

Significant performance tuning for:– FNDLOAD– AKLOAD

– Multi-Lingual Synch

Synchronize Multi-lingual tables in parallel

Tuning based on large volume upgrade tests

11i Patch Utility Enhancements (#1)

Checkfile feature– Activated with database drivers

– Records file and version information in the database

– Executes commands only if they have not executed previously for the highest onsite version of the file

– Eliminates need for package revision cache

Automatic job deferral– Re-tries failed jobs at the end of the phase

– Replaces auto restart

11i Patch Utility Enhancements (#2)

Unified Driver– Requires a single AD Patch run

AD Merge Patch improvements– Can merge all patch types and driver types– Characterset is converted during merge– Significant performance improvement

Automated Pre-requisite checking – AD Patch confirms prerequisite is satisfied– Allows for smaller patches, less overhead– Duplicate patch avoidance

11i Patch Utility Enhancements (#3)

Password Hider– Passwords automatically removed from log files

Manual Step Elimination– Automatically compiles Apps

– Automatically Maintains MRC

– Automatically compiles JSP

Technology Stack Validation Utility– Confirms minimum techstack requirements

– Refers to appropriate MetaLink notes

11i Patch Utility Enhancements (#4)

Distributed AD– Apply patches using all your middle tier horsepower to

significantly reduce downtime

adworker16…

adworker20

adworker11…

adworker15

adworker06…

adworker10

adpatchadworker01

…adworker05

Case Study

11.5.5(+) to 11.5.9

300 GB database

Languages: US English & German

Characterset: UTF8

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 3.5 hours 2.5 hours

Database update 20 hours 10 hours

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

Function US GermanReduction

File system update 3.5 hours 2.5 hours

Database update 20 hours 10 hours

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 20 hours 10 hours

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

3.5 hour reduction for file system update:reduced AD overheadfaster driver readprior charset conversion

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 20 hours 10 hours

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

3.5 hour reduction for file system update:reduced AD overheadfaster driver readprior charset conversion

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

12.5 hour reduction for database update:checkfile loader performancedirect PL/SQL increased parallelismreduced overhead reduced worker idle

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 9 hours

File Generation 3 hours 3 hours

Total 51 hours

12.5 hour reduction for database update:checkfile loader performancedirect PL/SQL increased parallelismreduced overhead reduced worker idle

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 3 hours 3 hours

Total 51 hours

8 hour reduction MultiLingual Synch:runs in parallelperformance improvements

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 3 hours 3 hours

Total 51 hours

8 hour reduction MultiLingual Synch:runs in parallelperformance improvements

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 2 hours 2 hours 35%

Total 51 hours

2 hour reduction file generation: parallel report generation less overhead

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 2 hours 2 hours 35%

Total 51 hours

2 hour reduction file generation: parallel report generation less overhead

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 2 hours 2 hours 35%

Total 25 hours 50%

26 hour total reduction.

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 14.5 hours 3 hours 40%

MultiLingual Synch n/a 1 hour 90%

File Generation 2 hours 2 hours 35%

Total 25 hours 50%

26 hour total reduction.

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 5 hours .5 hour 80%

MultiLingual Synch n/a 1 hour 90%

File Generation .5 hour .5 hour 85%

Total 10 hours 80%

Additional 15 hour reduction using distributed AD.

Function US GermanReduction

File system update 2 hours .5 hour 60%

Database update 5 hours .5 hour 80%

MultiLingual Synch n/a 1 hour 90%

File Generation .5 hour .5 hour 85%

Total 10 hours 80%

Additional 15 hour reduction using distributed AD.

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update none none 100%

Database update 5 hours .5 hour 90%

MultiLingual Synch n/a 1 hour 90%

File Generation none none 100%

Total 6.5 hours 87%

Additional 3.5 hour reduction using staged APPL_TOP.

Function US GermanReduction

File system update none none 100%

Database update 5 hours .5 hour 90%

MultiLingual Synch n/a 1 hour 90%

File Generation none none 100%

Total 6.5 hours 87%

Additional 3.5 hour reduction using staged APPL_TOP.

Case Study 11.5.5 to 11.5.9

Function US GermanReduction

File system update none none 100%

Database update 5 hours none 90%

MultiLingual Synch n/a none 100%

File Generation none none 100%

Total 5 hours 90%

Additional 1.5 hour reduction “hot patching” translation.

Function US GermanReduction

File system update none none 100%

Database update 5 hours none 90%

MultiLingual Synch n/a none 100%

File Generation none none 100%

Total 5 hours 90%

Additional 1.5 hour reduction “hot patching” translation.

11i Patch Process Enhancements

50% step reduction in 11.5.9

Recreate Grants and Synonyms

Compile Apps Schema

Maintain MRC

Compile Java Server Pages

Compile Flexfield Data ²

Gather Schema Statistics¹ ²

Generate Message Files ¹

Run multiple patch drivers¹

45% reduction in product steps¹

¹Maintenance Pack only

²Concurrent Request

50% step reduction in 11.5.9

Recreate Grants and Synonyms

Compile Apps Schema

Maintain MRC

Compile Java Server Pages

Compile Flexfield Data ²

Gather Schema Statistics¹ ²

Generate Message Files ¹

Run multiple patch drivers¹

45% reduction in product steps¹

¹Maintenance Pack only

²Concurrent Request

11i Quality Improvements (#1) Generated database drivers

Internal standards compliance checker– Automatically checks over 200 standards

SQL Repository– Parses over 400,000 SQL statements per release

– Explain plans created against 5 volume databases

Automated testing– WinRunner based infrastructure

– 400% increase in coverage

11i Quality Improvements (#2) Release Engineering process changes

– Earlier testing of

Product integration

Pseudo-translations

– Additional pre-release testing of

Translations (by countries)

OCS business flows

EBSO customer business flows

Internal production instances

Release Engineering Internal Early Access Program

11i Translation Improvements

Simultaneous release of all languages for:– Maintenance Packs– Family Packs/Minipacks

Translation escalation process– Usually available next day

Merged translations– Immediate availability with no translation regression

Translation quality improvements– 97% reduction in translation bugs 2001 - 2003

Configuration Management

OAMContext Editor

OAMContext Editor

AutoConfigAutoConfigRapid Install

config.txt

ApplicationsContext File

<sid> <host>.xml

ApplicationsConfiguration

Templates

GeneratedConfiguration

Files

jserv.propertiesappsweb.cfghttpd.conf

GeneratedDatabaseUpdates

APPS_WEB_AGENTICX_FORMS_LAUNCHERTCF:PORTTCF:HOST

Oracle Applications ManagerOAM Configuration Context Editor

Oracle Applications ManagerConfiguration History

Oracle Applications ManagerCompare and Restore Settings

Oracle Applications ManagerSearch Patches

Oracle Applications ManagerSearch Patches – Advanced

Oracle Applications ManagerPatch Summary

Oracle Applications ManagerFiles Copied by a Patch

Oracle Applications ManagerDatabase Driver Actions

Oracle Applications ManagerFile History

Oracle Applications ManagerPatch Advisor

Oracle Applications ManagerCreate Patch Filter

Oracle Applications ManagerView Advice

Oracle Applications ManagerView Patch Details

Oracle Applications ManagerView Patch Impact

Oracle Applications ManagerPatch Impact Details

MetaLink Patch Search 2.0Simple Patch Search

MetaLink Patch Search 2.0Product/Family Search

MetaLink Patch Search 2.0Saved Search

MetaLink Patch Search 2.0Advanced Search

MetaLink Patch Search 2.0Product Name Search

MetaLink Patch Search 2.0Product Hierarchy Search

MetaLink Patch Search 2.0Search Results

MetaLink Patch Search 2.0Saved Searches

MetaLink Patch Search 2.0Superseded Patch Details

MetaLink Patch Search 2.0Quick Links

MetaLink Patch Search 2.0Quick Links Mouseover

MetaLink Patch Search 2.0Server/Tools Quick Links

Maintenance Initiatives

MetaLink Patch Search

Improved flow to download patches

Automated notifications

Translation escalation/merge requests

Electronic Product Delivery

Install/Cloning Technology

Rapid Install “-techstack” option

– RDBMS 9.2.0.3 / 9.2.0.4

RAC Install

Complex cloning

– Multi-node to single-node

– RAC support

iSetup Integration

Patching Enhancements

Parallel index creation

In progress patch status

Improved patch reporting

Improved Patch Advisor

Additional impact analysis

Eliminate MRC schema

Concurrent AD sessions

CP A, BWeb A, BForms A, BAdmin

Multi-Node EnvironmentOne Admin NodeTwo Forms NodesTwo Web NodesTwo Concurrent Processing NodesNot using Shared APPL_TOP

CP A, BWeb A, BForms A, BAdmin

1) C driver on Admin node2) C driver on Forms node A3) C driver on Forms node B4) C driver on Web node A5) C driver on Web node B6) C driver on CP node A7) C driver on CP node B

8) D driver on Admin node9) G driver on Forms node A10) G driver on Forms node B11) G driver on Web node A12) G driver on Web node B13) G driver on CP node A14) G driver on CP node B

Standard AD

CP A, BWeb A, BForms A, BAdmin

1) C driver on ALL nodes simultaneously2) D driver on admin node and G driver on all nodes simultaneously

AD Concurrent Sessions

Linux Migration Utility

Quickly (~8 hours) and easily (~9 steps) migrate middle-tier to Linux

Can cut over with only a few minutes of downtime

Retains current patch level

Retains many customizations

Includes tech stack upgrade

Linux Migration Utility

Old System(Windows/Unix)

New LinuxMiddle Tier

ARU/MetaLink

Rapid InstallTechstack CDs

Generic files

Tech stack

Port-specific files

Long-term Initiatives

Fully Distributed AD

“Pre-downtime” patching

– Schema Clone to pre-stage PL/SQL and view changes

Granular linking & generation

Manual step infrastructure

AutoConfig Integration with OEM

For more Information…MetaLink InformationMetaLink Information Note IDNote ID

Cloning Oracle Applications 11iCloning Oracle Applications 11i 230672.1230672.1

Using AutoconfigUsing Autoconfig 165195.1 165195.1

Oracle Applications ManagerOracle Applications Manager 166762.1166762.1

Using a Shared APPL_TOPUsing a Shared APPL_TOP 233428.1 233428.1

Distributed ADDistributed AD 236469.1 236469.1

Using a Staged APPL_TOPUsing a Staged APPL_TOP 242480.1 242480.1

Linux Migration UtilityLinux Migration Utility 238276.1 238276.1

Maintaining Apps FAQMaintaining Apps FAQ ( (coming soon)coming soon)

MetaLink InformationMetaLink Information Note IDNote ID

Cloning Oracle Applications 11iCloning Oracle Applications 11i 230672.1230672.1

Using AutoconfigUsing Autoconfig 165195.1 165195.1

Oracle Applications ManagerOracle Applications Manager 166762.1166762.1

Using a Shared APPL_TOPUsing a Shared APPL_TOP 233428.1 233428.1

Distributed ADDistributed AD 236469.1 236469.1

Using a Staged APPL_TOPUsing a Staged APPL_TOP 242480.1 242480.1

Linux Migration UtilityLinux Migration Utility 238276.1 238276.1

Maintaining Apps FAQMaintaining Apps FAQ ( (coming soon)coming soon)

AQ&Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S