oracle open world september 28 – october 2, 2014 jim mckinstry oracle consulting practice director...
TRANSCRIPT
Oracle Open WorldSeptember 28 – October 2, 2014
Jim McKinstryOracle Consulting Practice Director
Bill CallahanDirector, Product and Technology
2
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
3
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
4
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Oracle Maximum Availability Architecture (MAA) TiersAvailability Service Levels for Unplanned and Planned Outages
BRONZE
SILVER
GOLD Comprehensive HA and Disaster ProtectionZero or near-zero data loss
High Availability (HA) for Recoverable Local Outages Data protected as of last backup
Single Instance Database, Basic Service Restart Data protected as of last backup
PLATINUM Zero Outage for Platinum Ready ApplicationsZero data loss
5
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Oracle Engineered Systems significantly simplify the architecture with pre-built and pre-tested configurations.Oracle Engineered Systems provide no single point of failure architectures.Oracle Maximum Availability Architecture ensures best practices to minimize or eliminate unplanned and planned downtime (including rolling upgrades).Oracle provides bundle patches that are pre-tested to minimize the risk of downtime.
MAA provides the performance and scalability to support CCC’s growth.
Oracle Engineered Systems: Increase availability Decrease TCO Increase scalability Increase performance
Benefits of MAA with Engineered Systems for CCC
6
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
AUTO PHYSICAL DAMAGE—INSURANCE
CASUALTY
EVALUATE AND REVIEW
first partybill review
third partybill review
RepairConsumer New Opportunities
REPAIR
INTAKE
dispatch
APD—Insurance IS & AIS Capabilities
intelligence
OPTIMIZE
auditworkflow
strategicparts
sourcingestimatingtablet
estimating
EVALUATE AND REVIEW
predictivecausation
injurycausation
INTAKE RESTORE AND RESETTLE
medicaladvisoryservices
injuryevaluation predictive
OPTIMIZE
analytics
partsprocurement
repairworkflow
REPAIR STATUS
status
predictive open shop
INTAKE
salvage
RESTOREAND
RESETTLE
consultingintelligence
OPTIMIZE
dispatchtotal loss
guided touraudit
workflownext gen
scheduling
strategicparts
sourcingestimatingtotal loss
next
EVALUATE AND REVIEW
Who is CCC?
8
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
9
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
CCC 2011 Initial Environment
Oracle/SUN M5000 and M4000 ServersCold Failover (4 hour RTS/? RPO)Oracle 10g Database Single Instance
Block SAN Storage30TB Combined storage across 20 databases45TB for Claim Folder/EDS
Performance IssuesRegular Performance disruptions caused by SAN Contention Reprocessing of customer transactions could
take as much as 24 hours
Customer Portal Experience – Poor Some queries related to search took 15-20
minutes – resulted in browser timeouts
10
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
CCC 2011 Initial Environment (cont’d)
Data WarehouseRedbrick (10TB On Disk Footprint)ETL/Reports 6+ hoursMicroStrategy – 30+ minutes
Web/Application LayerOracle/SUN SPARC servers (V490, E2900, M4000) Oracle 10g SOA/OC4J Applications
DR Using SAN ReplicationRepurposed Load Test Systems at DR Site for DR
11
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
12
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Exadata Implementation 2012Migrated – X2-2 Full Rack in Prod and
DRClaim Folder7 SOA Dehydration StoresData Warehouse8 other application databases
RAC for HAImplemented Services for quick failover
Storage reduction – 30TBImplemented Secure File LOBS
Reduced Application incidents by 60% where IO had been a bottleneck
Storage Offload (Automatic!)Portal Search < 30 seconds.
Data Warehouse1.8TB On Disk – With HCC!ETL/Reports < 1 hrMicrostrategy - < 5 min
DR Oracle Dataguard
Exadata X2-2 Full Rack
Exadata X2-2 Full Rack
13
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
DR – Exadata databasesDatabase recovery time for DR reduced from 4 hours to 30 minutes Recovery resources reduced from 3 DBA team members to 1 DBA
Patching Goal was to get to Platinum Support CertificationFull Frame (Storage Cells, CRS, RDBMS, IB) requiredPainful – requiring a 16 hour down time schedule Bug in CRS prevented rolling CRS Patching – Caused random nodes to
restart Reluctance to perform Cell Patching rolling – due to single frame in
production
Paid Off – It wasn’t for nothing! Control File Caching to Flash Cache – Huge relief in log file sync waits Significant buffer busy “global transaction manager” improvements
We need a local standby
Year 1 with Exadata
14
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Half Rack Exalogic980+ Application EARs Migrated off of OC4J
60% overcapacity for Redundancy
Eliminated 30 SPARC Servers in Production
Implemented SDP over Infiniband for communication to Exadata Databases
40% Improvement in application Performance
DR ZFS Storage ReplicationEnables rapid provisioning of Prod Images over Load Test nodes
Exalogic Implementation
15
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
16
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Local DataGuard Physical StandbyX3-2 Full Rack – Used for PrimaryImplemented in 30 days
RTS and RPO Improvements30 minute recovery (full frame switchover)<15 minutes RPO
Patching ImpactStandby first Patching optionAchieved Platinum Support Certification
Performance20% improvement in CPU utilizationEliminated remaining IO contentionBackups run from Standby (Must have Active DG)
DR Oracle Dataguard
Exadata Implementation 2013
Exadata X2-2 Full Rack Exadata X2-2 Full Rack
17
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Year 2-3 with Exadata
Platinum Services - PatchingDirectly coordinated patching with Platinum SupportPlatinum will make you plan, this is a good thing.
Patching First half was painful again, but in a different way– We had a schedule, but did not complete platinum fast enough to
schedule them!– They worked hard to support us, but the last minute details stung.– Local Standby masked all of it from our customers.
Second half (Happening right now!)– Planning since May– Plans have been set since 3 weeks after July Patch set– I am here, not there, which has to tell you something.
18
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future
Agenda
19
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Years ? with Exadata – What is next?
Storage! - We have grownClaim folder is 104TB now
Standbys Improving RTS– Considered Logical Standby – but too many unsupported datatypes.– Tuning Data Guard to bring switchover down to 10 min.– Automating steps – Our Offshore will execute our next DR
Weblogic Active Grid Link– Implemented in phases since January 2014– Should finish the across the board implementation as final apps
move to Weblogic
21
©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.
Thank You