oracle application upgrade

7/30/2019 oracle application upgrade

1/31

Twelve Steps to an Oracle Apps Upgrade

11.5.9 to 12.0.6 Upgrade Experiences from Hazelden Foundation

Jeff Vold John Stouffer

Hazelden Foundation Oracle E-Business ACEOracle Apps DBA Independent Consultant

[email protected] 214 535 6847

www.hazelden.org [email protected]

www.justadba.com


2/31

About Hazelden Foundation

Environment and Project Overview

12 Steps to an Oracle Upgrade

Other Observations

Questions


3/31

Since its 1949 founding in a Minnesota lakeside farmhouse,Hazelden has grown into one of the world's largest and mostrespected private not-for-profit alcohol and drug addiction

treatment centers.We have helped tens of thousands of people reclaim their livesfrom the disease of addiction. Hazelden offers the nations mostcomprehensive approach to addiction by addressing the full rangeof patient, family, and professional needs, including treatment andcontinuing care for youth and adults, research, graduate school ofaddiction studies, public education, and publishing.

We have facilities in Minnesota, Oregon, Illinois, New York, andFlorida. For more information, visit www.hazelden.org


4/31

Modules

Financials (AR GL AP FA PO)

Inventory (INV, BOM, Advanced Supply Chain)

Order Management (including Advanced Pricing and Quoting)

Components of CRM and Project Costing

iStore

Taxware - 3rd Party Tax Engine

Customizations

Custom Interfaces, code, and objects

Custom seeded packages


5/31

Three Tiers

Hardware - Sun Solaris 880s & 480s (2.8 OS)

DB Tier

9.2.0.8 Database - Raid 5 disk

8 gig RAM and 4 CPUs

Admin/Concurrent Managers Internal Apps tier

Load Balanced (11501 Cisco Content Switches)

Web, Forms, and Reports

Discoverer 10g (Note:373634.1 Option B)

External Apps Tier (In DMZ)

iStore Custom Website using JBoss

Disk Space Allocated (Production)

Database Space 198 gig/Used Space 132 gig

**Environment Space allocated 667 gig


6/31

Sun Solaris M5000 Servers (Solaris 10 OS)

Reduced physical number of servers from 9 to 4

Still three tiers

DB Tier

10.2.0.4 Database Raid 10 Disk

16 gig Ram and 6 CPUs

Admin/Concurrent Managers

Internal Apps tier

Load Balanced (11503 Cisco Content Switches)

Web, Forms, and Reports

Discoverer 10g (Note:373634.1 Option B)

External Apps Tier (in DMZ)

iStore

Custom Website Using JBoss

Disk Space Allocated (Production)

Database Space 283 gig/Used Space 207 gig

**Environment Space allocated 1170 gig (extra allocated for growth)


7/31


8/31

Staff

1 Project Manager/Apps BA (and now E-Bus Tax Expert)

1 Apps SysAdmin/Analyst

1 Discoverer Administrator (& SysAdmin backup)

1 Java/Web Developer

1 Senior Software Engineer

1.5 DBAs

12 Super Users (No Staff Augmentation)

Unix, Network, and Desktop Resources

Total Staff Hours 08 & 09 18,631

Minimal Budget Available for Outside Consultants

Technical DBA

iStore Specialist

Financials & OM Consultant

Total Consulting Hours 08 & 09 1,011


9/31


10/31

Dont Underestimate Size, Scope, and Complexity

Read and follow all the breadcrumbs

Documentation, Metalink, Blogs, and OAUG

Training

Network with Others

Start Team Building Decide on Architecture

Hardware Configuration

Upgrade Method

Other Details

Testing

Review and Update Super User Test Plans Prior to Project

Plan for heavier user testing at the beginning

Test Both (11.5.9) Migrated Data & Newly Created Data Thoroughly !!

Customizations - Less Issues Than Anticipated

Spent many hours reviewing/updating/correcting documentation


11/31

Organization

Naming Conventions

Build Documentation

Unix Directory Structure

Windows IT Directory Structure

R12 Environments

Patch, Dev, Test, Training, Production

Managed Gold Environment with Backups

Expect bumps in the road

Extra clones

Extra SRs

Use Oracle Support Configuration Manager

Extra Patches (111 patches)

Other issues


12/31

Advanced Preparation and Planning Outage Window

DBA Team

Run through many iterations of practice build

Optimize process to reduce manual steps & time

Detailed build Instructions SysAdmin, Web, & Developers

Same holds true practice and optimize documents

Super Users

Pre-upgrade shutdown plan

Post upgrade Setups, Validation, Input Orders, Startup

Work with Business (and IT) on Go-Live Dates

Create end-to-end Go-Live Weekend Plan


13/31

Installation Guide: Using Rapid

Install R12 B31295-07

Mount the Red Start Here cd

=> perl /mnt/cdrom/Disk1/rapidwiz/adautostg.pl

Check Rapid Installer Version

cd /Stage12/startCD/Disk1/rapidwiz

./RapidWizVersion

Download latest Rapid install startCD directory.

(MetaLink Note 549389.1 12.0.4 Patch 8582089 12.0.4.10)


14/31

Run Rapidwiz

./rapidwiz -servername hfdvd (Logical Host Name)

Select Install Oracle Applications Release 12

Rapidwiz errors on OCM Details Screen Refer to MetaLink Note 746666.1

Now fixed if using RapidWizVersion 12.0.4.10

RW-50004 Step 3 of 5 adrunias.sh

APPS Tech Stack Registration fails

Continue with install & then rerun failed command separately

Now fixed if using RapidWizVersion 12.0.4.10


15/31

Apply 10.2.0.4 Database Patch (6810189)

Apply Additional DB Patches (i.e., Latest CPU/PSU)

Oracle/Toad Issue Package names listed multiple times

Upgrade OracleAS 10g 10.1.3.4 (.5 is latest) MetaLink Note 454811.1 lists latest Oracle AS 10.1.3.x

Upgrade OracleAS 10g Forms and Reports

MetaLink Note 437878.1 lists latest upgrade info

OC4J Configuration Assistant Fails

Developer Suite 10g Builders Now Included


16/31

Upgrade Guide: Release 11i to Release

12.0.4 (E12011-02) / 12.1.1 (E16342-01)

Apply Pre-Upgrade Patches to 11i

11i.AD.I

TUMS (Report Lists Steps NOT to Apply)

SLA Pre and Post Upgrade (MetaLink Note 604893.1)

Run AD Prep Scripts (Patch 5726010)

Others

Create a Clone of Production to New Server

Clone 9.2.0.8 Oracle Software

Copy database and perform recovery as new name


17/31

Upgrade Database to 10gR2 Manually or use DBUA

10gR2 DB Prep Guidelines for R12.0.4 (MetaLink Note 403339.1)

Interoperability Notes 11i with DB 10gR2 (MetaLink Note 362203.1)

Complete Checklist for Manual Upgrades to 10gR2 (MetaLink Note 316889.1)

Database Connect Role now only contains Create Session

utlu102s.sql may differ from dba_registry (MetaLink Note 456845.1)

catupgrd.sql ORA-00904 Other XML Error

Recreate Plan Table (MetaLink Note 605317.1)

Data Mining and OLAP need to be installed

Had to rerun olap.sql and dminst.sql to be valid

Run Script to Fix Korean Lexers (dr$fnd_lobs_ctx$i max extents)


18/31

Review List of All Known Application Patches and Download

Use admrgpch to Group Patches

Prior to Patching

Review R12 Best Practices

Increase Tablespaces Modify $APPL_TOP/admin/adovars.env - Increase ADJREOPTS

Disable Custom Triggers

Other Fixes to Prevent adworker Errors

Enable Maintenance Mode with adadmin and Let it Rip !

12.0.4 Upgrade Driver E-Bus Tax Pre-Install

E-Bus Tax Data fixes

cd $AU_TOP/patch/115/driver (u6394500.drv)

adpatch options=nocopyportion, nogenerate portion

Useful Unix utilities - screen, less, & gvim


19/31

Run Pre-Install Merge Patch

Run Merge Patch (Includes 12.0.6)

12.0.6 long_running_jvm=false (MetaLink Note 761869.1)

One-off Patches Not Included in a Merge Patch.

Post Patching Steps

Use of Logical host names created challenges

Run fnd_conc_clone.setup_clean

SysAdmin Responsibility => Install => Nodes - Corrected host names andIP Addresses

Fix FND_CONCURRENT_QUEUES

1000 Concurrent Manager Programs

ARHMAILS - Mailstop Migration Runs 6.5 Hours

Support provided new arhmails.sql (no patch)


20/31

MetaLink Reference Note 384248.1 (Check Note 785296.1)

Two Methods for Creating

Using Rapidwiz (Section 3)

Not Using Rapidwiz (Section 4)

Share Mount Points from DB Server

Non-Production One NFS

Production Five NFS Mount Points

appl, comn, 10.1.2, 10.1.3, and inst

Run adpreclone on appsTier

perl adpreclone.pl appsTier

Encountered issue with cookies ( Notes 465583.1 and 268830.1)

Run adclonectx.pl to Add a Node

perl adclonectx.pl addnode contextfile=


21/31

Provide the values required for creation of the

new APPL_TOP Context file.

Target System Hostname (virtual or normal) [pra1] : pra1

Target System Domain Name : cc.hazelden.org

Target System Root Service [enabled] : Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] : disabled

Target System Other Services [disabled] :

Do you want to preserve the Display [prd:0.0] (y/n) ? : n

Target System Display [pra1:0.0] : localhost:15.0

Database port is 1529

Do you want the the target system to have the same port values as the sourcesystem (y/n) [y] ? : y


22/31

Autoconfig (MetaLink Note 387859.1)

Meant to make managing configuration easier

Review changes first using adchkcfg.sh

Running AutoConfig

Rolling back an AutoConfig Session using restore.sh

Custom Templates

view httpd.conf

## $Header: httpd_conf_1013.tmp 120.25.12000000.9

Or run adtmplreport.sh target=httpd.conf

$FND_TOP/admin/template/custom

Document all changes

Include files (IFILE)

example.. tnsnames.ora

IFILE=/u19//10.1.2/network/admin/DEV_hzdev_ifile.ora


23/31

Log into OAM with SYSADMIN

Go into License Manager

Click Products under License

Click License Component Application

Click Continue and Register Modules

Through adadmin

- Generate message files

- Generate forms files

- Generate report files

- Generate product JAR files and force generation Yes

- Compile APPS schema

- Compile menu information

- Compile flexfields

MetaLink Notes 577644.1 and 371016.1


24/31

Reapply Customizations

Create Custom Directories (Same convention as seeded)

Register custom tops (Navigation: Applications=>Register)

Migrate Code from 11i Top to New R12 Custom Top Install Custom and Custom Seeded Objects & Code into Database

Created script to check if custom seeded object were overwritten

Examples of post upgrade fixes

FSGs Completes with Error ORA-1722 (Note 579655.1)

R12: Tools -> Copy Function fails with FNDFS Error (Note 419142.1)

Cant Insert User Role When Assigning Responsibility (Note 559347.1)

frmcmp_batch.sh ora-12154 Couldnt resolve connect (Note 826024.1)

ORA-01001 When Clicking Cancel-Query Button [ID 436537.1]

Check for Invalid Objects, Statistics, Full Backup, Clones, and More


25/31

Upgrade Time

DBA backup to hand off for setups - 33 hours

SysAdmin/Super User Setups 13 Hours

Super User Non-Transactional Validation 3 hours

Super User Transactional Validation (Input orders) 6 hours

Super User/IT Transactional Troubleshooting 19 hours

Go-Live Decision? => Its a go!

Some Stats as of 4/1/2010

Internal Problem Requests

Total 589/Pre Go-Live 445/ Post Go-Live 144

R12 Oracle Support Service Requests

Total 235/Pre Go-Live 176/ Post Go-Live 59

Top Modules

E-BusTax, Receivables, Order Mgmt, Payments, & Payables

iStore not as bad as expected


26/31

Pre Go-Live SRs 7/27/08 7/27/09


27/31

Post Go-Live SRs 7/28/09 4/1/10


28/31

XML Reports and Logical Host Names

Java.Net.ConnectException: Connection Timed Out (MetaLink Note 753395.1)

Patch 7449694 and Profile option change

RUP6 APP-FND-01540: Unsuccessful logins since last session

11.5.10 Fix But Disables All Notifications

Oracle Provided Fix Patch 5015899

Credit Card Issues

Migrated Credit Card Orders not Masked (Patch 7708802)

Credit Card issues with back orders (still issue)

AP Suppliers and Bank Accounts (Now in TCA)

Thoroughly check bank accounts migrated correctly

External Bank account numbers missing (Oracle API to Update)


29/31

Tax Issues

E-Bus Tax Integrated with 3rd Party Taxing Solution Taxware

EBTax Many Upgrade Issues (Note 602030.1)

Multiple Issues with Tax Exemption migrations (Some fixed others not)

Taxware does not work for Canada

Need to use E-Bus Tax

Workflow Issues

After upgrading to 12.0.6, WFERROR is ending in Error (Note 815173.1)

Workflow error alerts when customer is created/updated (Note 1070881.1)

Workflow error alerts when creating Party Relationship for Org (Note 864353.1)

Disable Workflow AP email Remittance Advice not used (Note 472389.1)

Purge ICX_SESSION_ATTRIBUTES Issue

R12 ICX: Limit Time = 24 (hours) (Seeded with 999)

iStore

12.0.6 Lost ability for branded sign-in page (Note 732970.1)


30/31


31/31

Jeff Vold

[email protected]

www.hazelden.org

Hazelden Foundation

And

John Stouffer

Independent [email protected]

www.justadba.com

oracle application upgrade

Documents