oracle database patching best practices ii eleanor meritt, david price vice presidents oracle...

Oracle Database Patching Best Practices II

Eleanor Meritt, David PriceVice PresidentsOracle Product Development, Sustaining EngineeringSeptember 30, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |


Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

Preparing for Patching

Best Practices for Testing Patches

Applying Patches

Patching in the Cloud

Other Patching Related News

1

2

3

4

5


Preparing for Patching


Assessing Risk


One Off / Interim Patch

• Fixes a single bug: easy to verify if problem has been fixed, quickly available• Released with full component level regression testing at Oracle• Accessible by everyone with a support license• Low risk of introducing breakages• Vast majority can be installed with zero or minimal downtime• Drawback:

– Easy availability leads to tendency to customize environments by combining interim patches with other patches, causing supportability & maintainability problems

Risk Assessment


Security Patch Update (SPU or CPU)

• Quarterly Patch to fix security vulnerabilities• Extensively tested at Oracle• Some fixes can be quite involved and may require post installation steps or

configuration changes• Fix verification is by nature is very difficult• Historically has been very low risk• Drawbacks:

– Business wide coordination effort may need to be in place to manage fast roll-out.– Because the SPU does not contain fixes for high impact non-security bugs

encountered by customers, a mission critical system will almost certainly need to combine the SPU with one-offs, increasing overall risk.

Risk Assessment


Patch Set Update (PSU)

• Quarterly Patch to fix most recent high impact bugs• Contains Security Content• Extensively tested at Oracle• Strict content inclusion criteria:

– No optimizer changes, must be RAC rolling installable, DG Standby First Installable, fixes already tested by customers

– Low level volume of content

• Low risk• Drawback:

– Strict content inclusion restrictions can sometimes lead to need to customize environments with one-off patches as well.

Risk Assessment


Bundle Patch (BP)

• Quarterly Patch to fix high impact bugs for a given configuration (e.g. Exadata)– Contains PSU Content

• Extensively tested at Oracle• Content inclusion criteria to address stabilization needs of majority of

customers running this configuration:– Also optimizer changes, must be RAC rolling installable, DG Standby First Installable.

• A little higher risk than other patches• Drawbacks:

– Less restriction on content leads to higher volumes of fixes. Some fixes are getting released for the first time within the BP.

Risk Assessment


Risk Assessment

• Try to avoid combining patches. These increase risk due to untested combinations.

• Apply BP’s if you are running with engineered systems, Database In-Memory, or on Windows– PSU’s for everyone else

• Apply the latest PSU or BP on Upgrading to a new release• Ideally Patch Proactively every six months. • We know about patch problems usually within 4 weeks of release. Check

PAD on MOS for details on regressions & remedies

Summary Recommendations


And Finally on Risk

• All patches can be rolled back– Scripts are provided to undo SQL patch changes within patches.

• datapatch accomplishes this with 12c

– Binary patches can be undone by executing opatch(n)rollback or restoring the ORACLE_HOME from backup


Accessing Patches

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 14

Downloading patches with My Oracle Support (MOS)• Find the patch(es) by performing one of the following

– a Simple search for a patch or a group of patches– a Saved search– a Recent search– an Advanced search– a search using the Recommended Patch Advisor


MOS Patch Search Screen


MOS Conflict Checker

• A new conflict resolution tool is available from the patch Search screen• Self Service tool that doesn’t require an SR to be logged• Upload your OPatch inventory to resolve conflicts • Resolution patches that are available are provided immediately• Resolution requests are automatically filed if they do not exist

Document 1091294.1 "How to use the My Oracle Support Conflict Checker Tool"

https://support.oracle.com/DocumentDisplay?id=1091294.1


Patch Download


Testing Patches


A “How To” Guide for Testing Patches


Categorize Your Tests

• Functional Regression Tests– Purpose is to check if application flows and administrator activities behave as

expected.– These tests should be repeatable– Catalog all activities to test for and create tests to represent these.– Save expected results. – Run tests against the newly patched version. Compare new results against expected

results. – Divide tests into groups based on functional area so you can target runs depending on

patch.

The Major Categories


Categorize Your Tests

• Load Tests– Check how the system or application behaves under production load. – You may also want to simulate load beyond limits of normal operation for mission

critical systems.– Key outcome is that availability should not be affected.

• Performance Tests– Define key performance indicators to measure against for application flows. Examples

are responsiveness and throughput. – Measuring against these, no material negative differences should be seen.

The Major Categories


Testing Patches

• Real Application Testing (RAT) Capture / Replay functionality offers the ability to easily create load tests by capturing production system workloads and replaying them.

• RAT SQL Performance Analyzer (SPA) automates the process of assessing the effect of a patch on every SQL statement in your workload. It produces a report which you can analyze in order to remedy any potentially negative effect.

A plug for Real Applications Testing


Test Patches According to Risk

Risk Level Functional Testing Load Testing Performance Testing

Single One-off Patch Lowest Targeted Not Required Not Required

SPU Very Low Full Not Required Not Required

PSU Low Full Optional Not Required

Bundle Patch Slightly higher Full Recommended Optional

Any Combination of the Above Low to Medium Full Recommended Optional

33Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Applying Patches

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 34

Best Practices for Applying Patches• Aim for end to end automation of the patch application process. Write scripts to do this

if necessary.

• Ensure you have all prerequisites in place for patch application to complete without error

• If you have multiple patches to apply at a time, apply them in one downtime

• The vast majority of patches (>98%) can be installed in a highly available manner. Check the patch README for what is possible– RAC Rolling– Dataguard Standby First– Online patches: patch a running Database– All patches can be applied Out of Place : patch a Cloned ORACLE_HOME.

• Afterwards check that patch has installed correctly


Prerequisite Checking

opatch prereq CheckConflictAgainstOHWithDetail

• Check if the patch will conflict with already installed patches

opatch prereq CheckConflictAmongPatches

• Check if multiple patches will conflict with each other

opatch prereq CheckSystemSpace

• Check if there is enough system space to install the patch

opatch prereq CheckMinimumOPatchVersion

• Check the OPatch version against the required version for the patch.

OPatch/ocm/bin/emocmrsp

• Create a response file for OCM

–You may also want to run cvu and exachk where applicable


Optimizing patch application • Apply several patches during the same downtime• Unzip the patches in the same patch location• Execute opatch napply <patch_location> -skip_subset -skip_duplicate• Don’t mix bundle patches and overlays (ordering issue). Should apply BP

first, then overlays. Will be fixed in a later OPatch release.• napply applies multiple patches in the same session• skip_duplicates won’t apply patch if the patch is already on the

system• skip_subset won’t apply patch if the patch in the system already

contains all the fixes• In 12.1 datapatch takes care of applying the post SQL


Patch Installation based on Patch Packaging• System Patches

• ‘opatchauto’ automatically installs “System Patches”

– Install with Command : opatchauto apply

• Used for Exadata and GI bundles in RAC rolling Install mode

– Other options :

• opatchauto apply -nonrolling

• Singleton Patches• Used for singleton / interim patches

– Install with Command : opatch apply

– Other options :

• -silent : parameters are passed via the response file ‘-ocmrf’

• -local : Apply the patch on the local node of a RAC cluster


Running ‘datapatch’ with Database 12c• Ensures data/sql related changes are patched

• Takes care of installing/rolling back data changes

• Steps • After ‘OPatch apply’, Connect to the DB

• SQL> Connect / as sysdba

• SQL> startup

• SQL> alter pluggable database all open; [only for Multitenant DB]

• SQL> quit

• > $ORACLE_HOME/OPatch/datapatch [-verbose]

• Other options

– -apply / -rollback <patch_id>

– -force : runs apply/rollback as provided even if it was already done.

• Log location <oracle base>/cfgtoollogs/sqlpatch/<patch id>/<patch upi>. And the file name is <patch id>_[apply/rollback]_<dbname>[_<pdb name>].log.


What’s new in datapatch in 12.1.0.2

• No need to call catbundle.sql any more with PSU, BP installs– Datapatch is now ‘bundle-aware’ and takes care of installing specific bundles as

needed, using the dbms_sqlpatch package– Both bundle and non bundle patches are now only queryable via dba_registry_sqlpatch

– dba_registry_history is no longer used for patch information

• datapatch -rollbackall option is available to rollback all SQL patches currently installed


Checking Installed Patches via OPatch• OPatch Commands

– opatch lsinventory or opatch lsinv

• List the patches installed in the oracle home

• Other Options

– -all_nodes: Report the patches installed on given Oracle Home in all nodes of RAC system

– -detail(s): Display the components and the list of patches with their associated files

– -xml : Generate xml formatted output

– opatch lspatches

• List the installed patches and their description

• Other options:

– -bugs : Lists bug fixed by each patch

– -verify : Verifies if specified patch is installed in the oracle home


Checking Installed Patches via PL/SQL

Database 12c Queryable Patch Inventory• dbms_qopatch provides access to the OPatch

inventory information from within the database– PLSQL/SQL interface to view

• list of patches applied• check if a particular patch is applied• patch inventory across RAC nodes• SQL patch status

• Check datapatch entry in registry table : ‘select * from dba_registry_sqlpatch;’

GET_OPATCH_LIST

GET_SQLPATCH_STATUS

IS_PATCH_INSTALLED

GET_OPATCH_LSINVENTORY

DBMS_QOPATCH subprograms



• Cloud patching is all about scaling. Every part of the patching process must be considered in terms of scaling.

• Automation is crucial. Zero manual steps can be permitted when patching Cloud environments.

• Customization is the enemy of automation. All systems should be at uniform patch levels.

• Administrators must have complete confidence that a patching exercise has been successful. Logging & diagnostics are very important.

• Scale adds to overall risk, so testing strategy has to aim for comprehensive testing coverage, automating as much of the testing as possible.

Basic Principles



• Start with ensuring you can automate patching a single system end to end.• Push out the patching related changes from the base patched system to the

entire Cloud. There are various alternatives including IT automation software like Puppet or Chef. Oracle Enterprise Manager has some capabilities. Oracle Rapid Home Provisioning is newly available.

• Best practices for Cloud patching can be easily applied to single systems. Not necessarily so the other way round!

• The goals of Cloud patching can be met with all supported versions of the Oracle Database. However, 12c does have some extra features to make Cloud patching easy.

Best Practices


Database 12c Features to Ease Cloud Patching


Multitenant for Simplified PatchingPatching the Container Database Results in Patching all of its Pluggable Databases

Patch Container Database

12.1

12.X

12.X

12.1

ERPERP

CRM

Multitenant Container Database

ERP

CRM

12.X

12.1

ERPERP

CRM


ERP

CRM

12.X

12.1

ERPERP

CRM


ERP

CRM

Shutdown

Startup

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential

Patching Using the Multi-Tenancy Feature - Best Practices

)

•When plugging into a new container a PDB may have violations due to

• Database version (12.1.0.1 vs 12.1.0.2)• SQL patch mismatches• Database parameter mismatches such as

character sets or block size•The dbms_pdb.describe and dbms_pdb.check_plug_compatibility APIs can be used to determine if a given PDB can be plugged in successfully to a target container

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential


)

Preparation:

1) Create PDB description XML file for PDB(PDB1) in question: exec dbms_pdb.describe (‘PDB1_Unplug.xml’, ‘PDB1’);2) In the target container environment, check plug compatibility begin if dbms_pdb.check_plug_compatibility('PDB1_Unplug.xml', ‘PDB1') then dbms_output.put_line(‘no violations found'); else dbms_output.put_line(‘violations found'); end if; end;

Plugin compatibility issues, if any, will be reported in pdb_plug_in_violations view



SQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 /no violations found

PL/SQL procedure successfully completed.

SQL> SELECT type, message, action 2 FROM pdb_plug_in_violations 3 WHERE name = 'PDBORCL';

no rows selected

Oracle Confidential

Scenario 1 – No plug in violations


Patching Using the Multi-Tenancy Feature - Best PracticesSQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 /violations found


TYPE MESSAGE--------- --------------------------------------------------------------------------------ACTION--------------------------------------------------------------------------------ERROR PSU bundle patch 1 (PSU Patch 12345): Installed in the CDB but not in the PDB.Call datapatch to install in the PDB or the CDB

Oracle Confidential

Scenario 2 – SQL patch present in target container but not in source container



SQL> BEGIN 2 IF dbms_pdb.check_plug_compatibility('/tmp/PDBORCL.xml') THEN 3 dbms_output.put_line('no violations found'); 4 ELSE 5 dbms_output.put_line('violations found'); 6 END IF; 7 END; 8 /violations found


TYPE MESSAGE--------- --------------------------------------------------------------------------------ACTION--------------------------------------------------------------------------------ERROR PSU bundle patch 1 (PSU Patch 12345): Installed in the PDB but not in the CDB.Call datapatch to install in the PDB or the CDB

Oracle Confidential

Scenario 3 – SQL patch present in source container but not in target container


Multitenant Plug/Unplug

Scenerio Recommended Action

1: SQL Patches in both source and target container None needed – safe to plug in

2: SQL Patches in target container only Run datapatch in target after plug in

3: SQL Patches in source container only Run datapatch -rollback <patch id> –force [–bundle_series] in source before unplug

Oracle Confidential

Scenario Summary


DB Cloning using EM12c Provisioning- Deploy Gold Images to the Cloud

• Mass Deployment of Oracle Software (Database, Real Application Clusters)

• Supports all versions up to 12.1 including Pluggable Databases

• Gold Image cloning and standardized software deployment via Profiles

• Lock down access for controlled and error free deployments

DB Provisioning

Source DB systems Target DB SystemsSoftware Library Storage

Save Gold image (and optionally data) from source systems to EM software library

Deploy saved Image and data to target systems with customizations


EM12c Patch Management Solution

Patches, Upgrades complete Database product familyProvides proactive Oracle recommendations (CPUs, PSUs,..)Simplified patching flow using Patch PlansComprehensive pre-flight checks and conflict resolution Support Out of Place, Rolling options for reduced/zero

downtime and rollback/switch backMass automation - multiple targets with multiple patches in a

single downtime Extensible framework, Patch Reports and “EMCLI” scripting

option

*Current support available for Databases only.** DBaaS on-premise / private cloud

End to End Patch Automation Solution for Oracle Databases


New - Oracle Rapid Home Provisioning

• Eliminate the need to patch individual Databases• Update any number of Databases with a single command• Ensure standardization through gold image lineage• Create reference homes on Centralized Home Server

– Apply patches once on Home Server– Distribute or update on-demand to the Cloud

• Fast and Efficient

Automating Patching for Cloud


Rapid Home Provisioning - commands• Create a gold image from

– An installed home– An existing workingcopy

• Create workingcopies from a gold image; optionally create database• Add a database to a workingcopy• Move an existing database to a different gold image

– Individually or all databases configured to this image– Optionally select non-rolling

• Manage roles and ACLs• Manage an RHP server and an RHP client


RHP Functionality Rollout

Oracle Grid Infrastructure 12.1.0.2:• RHP Server and Client• Support for Database Templates

• Home, Configuration, Data• NFS Mount and Local copy• Efficient storage• Provision and patch• User initiated distribution

• Push or pull• Support for Oracle DB

• Full workflow • Support for generic S/W images• Local change control

Coming Soon:– Support for GI homes– Generic image workflow– Remote home servers– Local changes tracking– Policy based rollout– Database upgrades– Oracle application templates– VM templates



• The dbms_rolling package, introduced in Database 12.1.0.2, enables automation of minimal downtime patching in a Dataguard environment.

• Patch a “Leading Group” - the new Primary Database(s), and switch over applications from the “Trailing Group” – the actual Primary Database, to point to the “Leading Group”

• Includes:– Preparation– Validation and planning– Start Phase– Finish



• Introducing the “Database Patch for Engineered Systems and DB In Memory” (DBP ENG/IM) with 12.1.0.2.– This patch is a super set of the PSU. It replaces the “Database Bundle Patch for

Exadata”. It is intended to be consumed by customers using the Exadata, Exadoop, ZDLRA, Big Data Appliance and In Memory Database features.

• 2 year error correction grace period for patch sets on the R2 release trains.– Terminal patch set has always been supported through the end of Extended Support

• Extended Support for Database 11gR1 ends August 2015• Premier Support for Database 11gR2 ends January 2015. First year of

Extended Support is at no additional cost.


References

• PSU known issues MOS note: 12274431.1• How to use the MOS Conflict Checker: 1092924.1• Document 1585822.1 “Database 12c Post Patch SQL Automation “

https://support.oracle.com/DocumentDisplay?id=1585822.1


Related Sessions

• How and why to Migrate from Schema Consolidation to Pluggable Databases CON7649, – Wednesday, Oct 1st, 11:30 AM - 12:15 PM - Moscone South – 306

• Databases to Oracle Exadata: The Saga Continues for Oracle Enterprise Manager–Based Patching – Wednesday Oct 1st 10:15 AM - 11:00 AM Moscone South - 300 CON8121

• Rapid Home Provisioning: Deploying and Updating Database Templates in a Cloud [CON8176] – Thursday, Oct 2nd, 9:30 AM - 10:15 AM - Moscone North - 131

oracle database patching best practices ii eleanor meritt, david price vice presidents oracle...

Documents

oracle andor

risk slide

oracle accessible

patching slide

sole discretion of oracle

security content

overall risk

low risk drawbacks