oracle goldengate performance tuning and …...• note 1448324.1 ogg integrated healthcheck script...
TRANSCRIPT
Oracle GoldenGate Performance Tuning and Monitoring Ease of Manageability
Volker Kuhr Senior Principal Product Manager Jiahui Shao Product Manager
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate and Data Integration
Energy/Industrial Insurance / Health Retail Other
Communications Finance / Banking Media Services Over 10K Customers Worldwide
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate Architecture
Bi-Directional
Capture Trail Files Network Firewall Cloud
Trail Files Delivery Database
Database
3
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Automatic Heartbeat Functionality
Oracle GoldenGate Diagnosis
1
2
4
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Automatic Heartbeat Functionality
Oracle GoldenGate Diagnosis
1
2
5
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New 12.2 Feature - Automatic Heartbeats
6
Built-in Functionality to Monitor End-to-End Replication Lag
Simple Automatic • Enable via a single “ADD
HEARTTBEATTABLE” command at each site
• Automatically discovers replication flows for all topologies
–Unidirectional –Bidirectional –N-way
Real-time • Database views to query
end-to-end real-time latency
• Lag information on incoming flow to and outgoing flow from a database
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Simple Bidirectional Replication Example • Enable Heartbeat functionality by executing GGSCI command ‘ADD
HEARTBEATTABLE’ at each database – Creates required heartbeat tables, views and jobs – Updates heartbeat every 60s by default
7
ExtA
ExtB
PmpAB RepAB
RepBA PmpBA
In Database A: GG_LAG View contents
LOCAL_ DATABASE
REMOTE_ DATABASE INCOMING_PATH INCOMING_
LAG
INCOMING_ HEARTBEAT_
AGE OUTGOING_PATH OUTGOING_
LAG
OUTGOING_ HEARTBEAT_
AGE
A B ExtB==>PmpBA==>RepBA 1.066182 14.150614 ExtA==>PmpAB==>RepAB 1.391718 25.387458
Database A Database B
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Monitoring Lag using GG_LAG View
8
Column Name Data type Description
LOCAL_DATABASE VARCHAR2(30) Local database name
CURRENT_LOCAL_TS TIMESTAMP(6) Current timestamp in UTC time zone
REMOTE_DATABASE VARCHAR2(30) Remote database name
INCOMING_PATH VARCHAR2(4000) Group names on the incoming flow
INCOMING_LAG NUMBER Period of time between remote database generating heartbeat and local database receiving heartbeat
INCOMING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus most recent heartbeat timestamp of remote database
OUTGOING_PATH VARCHAR2(4000) Group names on the outgoing flow
OUTGOING_LAG NUMBER Period of time between local database generating heartbeat and remote database receiving heartbeat
OUTGOING_HEARTBEAT_AGE NUMBER CURRENT_LOCAL_TS minus recent heartbeat timestamp of local database
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• GG_LAG_HISTORY View – Historical heartbeat lag
• GG_HEARTBEAT and GG_HEARTBEAT_HISTORY Tables – Underlying tables for the views – Get lag for each process on the path
9
Additional Tables and Views
Column Name
LOCAL_DATABASE
CURRENT_LOCAL_TS
REMOTE_DATABASE
INCOMING_PATH
INCOMING_LAG
INCOMING_HEARTBEAT_AGE
OUTGOING_PATH
OUTGOING_LAG
OUTGOING_HEARTBEAT_AGE
Column Name
INCOMING_HEARTBEAT_TS
INCOMING_EXTRACT_TS
INCOMING_ROUTING_TS
INCOMING_REPLICAT_TS
OUTGOING_HEARTBEAT_TS
OUTGOING_EXTRACT_TS
OUTGOING_ROUTING_TS
OUTGOING_REPLICAT_TS
…
} Extract lag } Pump lag } Replicat lag
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Discovers New Replication Flows Automatically
• Simply execute ADD HEARTBEATTABLE for new site
10
Example: Add new site Database C
In Database A: GG_LAG View contents LOCAL_
DATABASE REMOTE_ DATABASE INCOMING_PATH OUTGOING_PATH
A B ExtB==>PmpBA==>RepBA ExtA==>PmpAB==>RepAB
A C ExtC==>PmpCA==>RepCA ExtA==>PmpAC==>RepAC
Database A Database B
Database C (Newly added site)
ExtA==>PmpAB==>RepAB
RepBA<==PmpBA<==ExtB
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Diagnosing Replication Issues using Heartbeat Table
11
Diagnosis Cause
INCOMING_LAG is high
OUTGOING_HEARTBEAT_AGE is high
INCOMING_HEARTBEAT_AGE is high
OUTGOING_LAG is high
Action • Replication processes on the incoming path have stopped • Network issues • Heartbeat scheduler job has stopped
• Check processes on incoming path and the remote heartbeat scheduler job
• Replication processing is slow on the incoming path
• Use GG_HEARTBEAT table to determine slow process on the incoming path
• Replication processes on the outgoing path have stopped • Network issues • Heartbeat scheduler job has stopped
•Check processes on outgoing path and the local heartbeat scheduler job
• Replication processing is slow on the outgoing path
•Use GG_HEARTBEAT table to determine slow process on the outgoing path
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Heartbeat Functionality
Oracle GoldenGate Diagnosis
• GoldenGate Extended Metrics
• Integrated Healthcheck
1
2
12
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Potential Sources of Lag • System Resources
– Memory – CPU – IO and Network
• Configuration – Database – GoldenGate
13
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Tools for Performance Analysis
LogMining Server
Extract Pump
Replicat Inbound Server
• HealthCheck • UTL_SPADV • AWR & ASH
• HealthCheck • UTL_SPADV • AWR & ASH
GoldenGate Extended Metrics
OS Utilities (netstat)
OS Utilities (top, vmstat, iostat)
OS Utilities (top, vmstat, iostat)
14
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New GoldenGate Extended Metrics
• Access to Monitoring Point through Restful Web Services http://<hostname>:<mgr_port>/mpointsx
• Real-time insight into GoldenGate client programs
• Requires “ENABLEMONITORING” in GLOBALS
15
Fine-grained Performance Monitoring
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
New Extended Metrics • Status and Configuration Information • Process Level Metrics for Extract, Pump and Replicat • Thread Level Metrics for Extract and Replicat • Database Statistics for Extract and Replicat
– In-flight-transactions and Cache- & Queue statistics for Extract – Table statistics for Replicat
• Network Statistics
16
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
17
https://java.net/projects/oracledi/downloads/download/GoldenGate/OGGPTK.jar Graphical real-time instance monitoring
Replication Components
Process Status Change
Monitoring Metrics
Recording Function
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Integrated Healthcheck Report
• Comprehensive snapshot of the database components of Integrated Extract and Apply.
• Best Practices: Run periodically for differential analysis • Used for diagnostic purposes
– Configuration Issues – Performance
• Analyze at component and sub-component level
Snapshot of all GoldenGate related Information within the Database
18
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Healthcheck Script
1. Database configuration
3. Extract/Replicat Configuration
4. Extract/Replicat Statistics
2. Summary and Notification
19
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Notification Section
• Pre-Analysis with INFO, WARNING & ERROR Messages
• Heartbeat Table
20
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Record changes
Capture changes
Redo
Integrated Extract
Oracle Database
LCR1 LCR2 LCR3 …. …. ….
Logical Change Records
Extract Trail Files
Lag within the Integrated Extract
21
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
GoldenGate Logmining Server Architecture
Extract
• Requests LCRs from logmining server
• Performs Mapping and Transformations
• Writes Trail File
Logmining Server (Database Capture Process)
• Reader: Reads logfile and splits into regions
• Preparer: Scans regions of logfiles and prefilters based on extract parameters
• Builder: Merges prepared records in SCN order
• Capture: Formats Logical Change Records(LCRs) and passes to GoldenGate Extract
Reader Capture Builder
Logmining Server
Preparer N
Preparer 2
Preparer 1
Oracle Database
Integrated Extract
Trail Files
22
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Monitoring Integrated Extract
PATH 2 RUN_ID 15 RUN_TIME 2014-FEB-21 22:32:17 CCA N <C> OGG$CAP_EXT_1A 305260 305245 790813 LMR 0.0% 53.3% 33.3% "CPU + Wait for CPU" proc %idle %flowctrl %TopWait “Event” LMP(2) 0.0% 0.0% 146.7% "CPU + Wait for CPU" LMB 73.3% 0.0% 6.7% " " CAP 20.0% 0.0% 73.3% "CPU + Wait for CPU" <Q> "OGGADMIN"."OGG$Q_EXT_1A" 0.01 0.01 0 <E> EXT_1A 304027 80076708 790813 20% 6.7% 73.3% "CPU + Wait for CPU" <B> NO BOTTLENECK IDENTIFIED
• % of Idle, Flow Control and major Wait Event for each sub component • Check output for multiple occurrences over time • Increase Parallelism of Capture – if CPU is available
(LMR in flow control, and LMP is 0% idle with high CPU)
Streams Performance Advisor (UTL_SPADV)
23
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• State and "Ready to send Chunks" as one Indicator for an Extract program issue
• If "Ready to send Chunks" >> " Delivered Chunks" and Capture State indicates " Waiting for Client Requests", Extract program needs to checked.
• Check Extended Metrics and TRANLOGOPTIONS • BUFFER_SIZE , _READAHEADCOUNT, ASYNCTRANSPROCESSING …
Lag between Database and Extract client
24
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Integrated Replicat Replicat: Apply section
Trail Files
Replicat Lightweight Streaming
API Receiver Preparer Coordinator
Inbound Server
Applier N
Applier 2
Applier 1
Oracle Database
• Waiting for memory might indicate Memory issues Replicat IntegratedParams MAX_SGA_SIZE
25
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
• Total LCRs with Dependencies indicates transaction dependencies • One reason for Total LCRs with WM Dependency might be a large transaction
Integrated Replicat Transaction Dependencies
… … … … … … … … … …
20000
• Increasing EAGER_SIZE from Default to a moderately higher number
20000
26
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Apply Server Statistics – Wait Event Analysis
• Wait Event Analysis of Apply Server Processes – “Classical Waits”: Enqueues, IO Waits, … – Replication related Wait Events
• Check IO Subsystem, Classic Database Tuning (e.g.: INITRANS, Index Rebuild, …)
Apply Server Processes as common Database sessions
(wait on CPU /on CPU)
(wait on CPU /on CPU)
REPL Apply: dependency
27
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Additional Database Resources AWR & UTL_SPADV • . • UTL_SPADV Package to monitor database background
processes for Extract and Replicat – Two ways of displaying information: single line or HTML – Start and Stop monitoring – Shows bottleneck based on database wait events for the interval
28
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Key Performance Tuning Parameters • Database
– STREAMS_POOL_SIZE
• Integrated Extract – TRANLOGOPTIONS
• BUFSIZE • ASYNCTRANSPROCESSING • _READAHEADCOUNT
– TRANLOGOPTIONS INTEGRATEDPARAMS (…)
• MAX_SGA_SIZE • PARALLELISM
29 •
• Integrated REPLICAT – DBOPTIONS
• BATCHSQL • GROUPTRANSOPS (if PARALLELISM=1)
– DBOPTIONS INTEGRATEDPARAMS (…)
• MAX_SGA_SIZE • PARALLELISM • MAX_PARALLELISM • COMMIT_SERIALIZATION • EAGER_SIZE
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Summary • Monitoring:
– Use Automatic Heartbeat Table – Run periodically Healthcheck Reports
• Analysis – Use Heartbeat Table to drill down to the component – Monitor OGG extended Metrics – Healthcheck Reports – UTL_SPADV – Check Database Performance (AWR & ASH) and OS Performance
30 •
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
References
• MAA white paper Oracle GoldenGate Performance Best Practices http://www.oracle.com/technetwork/database/availability/maa-gg-performance-
1969630.pdf
• Note 1557031.1 Recommended patches for Integrated Capture/Replicat • Note 1448324.1 OGG Integrated Healthcheck Script • Note 1488668.1 GoldenGate Performance Data Gathering • Note 1596493.1 GoldenGate Integration with AWR • Note 1485620.1 Best Practices Downstream Capture
MOS Notes
31