an exadata database machine setup version doag

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2014 © Trivadis

An Exadata Database Machine Setupfrom POC to Production…

Umut Bozkurt

Trivadis AG

Senior Consultant

14.05.2014DOAG Exaday: An Exadata Database Machine Setup

1

2014 © Trivadis

Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution-Engineering und der Erbringung von IT-Services mit Fokussierung auf und Technologien im D-A-CH-Raum.

Unsere Leistungen erbringen wir aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden BetriebIhrer IT Systeme.

Unser Unternehmen


B E T R I E B

2014 © Trivadis

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort

3

12 Trivadis Niederlassungen mitüber 600 Mitarbeitenden

200 Service Level Agreements

Mehr als 4'000 Trainingsteilnehmer

Forschungs- und Entwicklungs-budget: CHF 5.0 / EUR 4 Mio.

Finanziell unabhängig undnachhaltig profitabel

Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden

Stand 12/2013

3


Hamburg

Düsseldorf

Frankfurt

FreiburgMünchen

Wien

BaselZürichBern

Lausanne

3

Stuttgart

Brugg

Willkommen bei Accarda

Spezialist für Bezahllösungen und Kundenbindung

Kundenkarten mit und ohne Zahlfunktion gehören seit über 40 Jahren zu den

Kernkompetenzen der Accarda; diese werden um Know-how und Expertise für

Bonusprogramme, Geschenkkarten, CRM, BI und Direct Marketing ergänzt.

Als Komplettanbieter mit einer leistungsstarken Infrastruktur konzipiert und betreibt Accarda

auch das Processing, Risikomanagement inkl. Bonitätsprüfung und Fraud Protection,

Debitoren- und Finanzmanagement sowie ein kundenfreundliches Call Center.

Kundenkarten

Geschenkkarten

Bonusprogramme

Mobile Payment

Molile Couponing

Direct Marketing

BI Services

Risk Management

© Accarda AG | Birkenstrasse 21 | 8306 Brüttisellen | www.accarda.com

2014 © Trivadis

AGENDA

1. Proof Of Concept

2. Storage Reorganization

3. Network Setup

4. High Availability

5. Migrating to Exadata

6. Q&A


5

2014 © Trivadis

Proof Of Concept

6


2014 © Trivadis

Environment

34 Servers (Linux on ESX, HP-UX)� 121 CPUs� 162 Cores

82 DBs� 25 Prod� 16 Pre-Prod� 17 Integration� 21 Development� 3 Test

7


2014 © Trivadis

Proof Of Concept

Trivadis Exadata X3 setup on site

Performance Tests� Export / Import� RMAN Clone� Batch Jobs / ETL / Reports

HA, Network and Storage concept was not part of POC

8


2014 © Trivadis

Sizing

Sizing by Oracle � Half Rack

� 3 Compute Nodes each 2x6 Cores, 256 GB RAM � 3 Storage Servers

� Quarter Rack� 2 Compute Nodes each 2x6 Cores, 256 GB RAM� 3 Storage Servers

Total 60 Cores => 30 CPU license

Leased from Oracle for 3 years. (Max. 5 years)

9


2014 © Trivadis

Storage Reorganization

10


2014 © Trivadis

Exadata Storage

Disks 1-2 Cell Server OS

Disks 3-11 DBFS_DG Disk Group � 1 TB Raw� ASM spfile, OCR, Voting

Each Storage Server is an ASM Failure Group


11

2014 © Trivadis

ASM Best Practice

ASM Best Practice: One failure group of space free

� Online Reorganizations

� Double Failures

12

NAME TOTAL_MB FREE_MB ONE_FAILURE_GROUP USABLE_FREE TOTAL_USABLE

DATA01 74.2 68.62 24.73 21.94 24.73

DBFS_DG 0.99 0.99 0.33 0.33 0.33

RECO99 22.83 22.35 7.61 7.37 7.61

TOTAL 98.02 91.96 32.67 29.64 32.67

select name,round(total_mb/1024/1024,2) total_mb,round(free_mb/1024/1024,2) free_mb,round(required_mirror_free_mb/1024/1024,2) one_fail ure_group,round( usable_file_mb /1024/1024,2) usable_free,round((total_mb-required_mirror_free_mb)/2/1024/102 4,2) total_usable

from v$asm_diskgroup ;


2014 © Trivadis

Exadata Initial Storage Setup

Exadata Setup: FRA / DATA disk group storage ratio?

� 20% to 80% => Backup on External Devices

� 60% to 40% => Backup on FRA

Resize online?

� YES (MOS:1465230.1)

13


2014 © Trivadis

Online Storage Reorganization

80% Data, 20% FRA => 70% Data, 30% FRA


14

2014 © Trivadis


First Rebalance of +DATA

� Shrink ASM Disk


15

alter disk group +DATA resize all size … rebalance power 32 ;

2014 © Trivadis


Second Rebalance of +DATA

� Drop ASM disks and Grid Disks on the first cell

16

alter disk group +DATA drop disks in failgroup… rebalance power 32

cellcli –e drop griddisk all harddisk prefix=…


2014 © Trivadis


Third rebalance of +DATA

� Add Grid and ASM Disks

17

cellcli –e create griddisk all harddisk prefix=… size= …

alter disk group +DATA add disk … rebalance power 32


2014 © Trivadis


Fourth, Fifth, Sixth, Seventh rebalance of +DATA

� Repeat Drop / Create for all Storage servers.

� 2 Failure Group of free storage: Drop/Create can be combined to reduce number of reorgs

18


2014 © Trivadis


First rebalance of +RECO

� Resize Grid Disks / ASM Disks

19

cellcli –e alter griddisk … size=…

alter disk group +FRA resize all size … rebalance power 32


2014 © Trivadis


For 3 cell servers

� Free Space: 2 x (cell size + % to be moved)� Disk group to be shrink: 4 Rebalance� Disk group to be extended: 1 Rebalance

� Free Space: one cell size + % to be moved� Disk group to be shrink: 7 Rebalance� Disk group to be extended: 1 Rebalance

Flexible configuration required� We will not have one failure group size of free space� 7 x Moving 40TB online

20


2014 © Trivadis

Storage Configuration

DBFS_DG left intact

� 17 Grid Disks => 17 ASM Disks� No storage wastage 4MB AU

� +DATA => 13 ASM Disks

� +FRA => 4 ASM Disks

Benefits:

� Flexible: 5.8% (=5.8 TB Raw) storage can be moved with a single reorg.

� New disk group can be created

21


2014 © Trivadis

Drawbacks

� Innermost tracks are slower than outermost tracks� Violating ASM best practice: All disks in a disk group should have the

same performance: I/O performance is constrained by the slowest speed drive.

� ASM striping on the same physical device.

22


2014 © Trivadis

Performance

23

Group Name Interval_mins reard_time write_time IO_time reads writes IO reads(ms) writes(ms) IO(ms)

+DATA GD_01 1 15.846727 1062.280839 1078.128 1026 976 2002 15.445 1088.402 538.525

+DATA GD_02 1 2.552395 1008.283094 1010.835 1051 803 1854 2.429 1255.645 545.218

+DATA GD_03 1 2.293635 1077.035056 1079.329 655 1065 1720 3.502 1011.301 627.517

+DATA GD_04 1 6.808609 1185.655312 1192.464 2241 1224 3465 3.038 968.673 344.145

+DATA GD_05 1 3.933126 912.725269 916.658 1302 974 2276 3.021 937.09 402.750

+DATA GD_06 1 1.657625 918.965771 920.623 1023 907 1930 1.62 1013.193 477.007

+DATA GD_07 1 1.997571 1037.79631 1039.794 1269 1050 2319 1.574 988.377 448.380

+DATA GD_08 1 1.370921 1086.59526 1087.966 827 938 1765 1.658 1158.417 616.411

+DATA GD_09 1 1.943523 1208.215758 1210.159 1160 967 2127 1.675 1249.448 568.951

+DATA GD_10 1 1.301434 1144.512351 1145.814 650 891 1541 2.002 1284.526 743.552

+DATA GD_11 1 2.245439 1126.597974 1128.843 2168 1026 3194 1.036 1098.049 353.426

+DATA GD_12 1 1.405714 941.956325 943.362 710 1310 2020 1.98 719.051 467.011

+DATA GD_13 1 2.002054 913.93025 915.932 423 1015 1438 4.733 900.424 636.949

+RECO GD_14 1 1.812914 132.248082 134.061 169 401 570 10.727 329.796 235.195

+RECO GD_15 1 2.27788 75.561454 77.839 540 1075 1615 4.218 70.29 48.198

+RECO GD_16 1 2.396498 0.254019 2.651 795 660 1455 3.014 0.385 1.822

+RECO GD_17 1 1.88004 27.146179 29.026 160 255 415 11.75 106.456 69.942

IO(ms) = (∆ reads + ∆ writes) / (∆ read_time + ∆ write time)

reads, writes : I/O request


2014 © Trivadis

Conclusion

Flexibility vs Performance

� Flexibility wins

Before sizing consider reorg possibilities

Open Questions:

� Striping on the same disk performance impact unknown

� Performance difference between innermost and outermost ASM disk is unknown

=> Further tests has to be performed

24


2014 © Trivadis


Network Setup

25

2014 © Trivadis

Shared Environment

(Scan-) Listeners allow to connect to any registered services.

Any service on any database can be connected with username/password.

26


2014 © Trivadis

Virtual LAN Tagging on Exadata

For VLAN Configuration MOS 1423676.1

Our configuration would be:

� One VLAN for each DB� 30 VLANs � 30 Listeners� Client setup if more than one

DB Service will be accessed

27


2014 © Trivadis

Oracle Connection Manager

Offers

� Connection multiplexing

� Access control

� Source routing

� Multiprotocol Support

� Firewall Proxy Support

Installed on a separate machine(s for HA)

No additional license for the machine running Oracle Connection Manager.

28


2014 © Trivadis

Oracle Connection Manager

Net Service Name

29

DB01_RW=(CONFIGURATION=(ADDRESS=(PROTOCOL=tcp)(HOST=CMAN_HOST)(PORT=1521))

(RULE_LIST=(RULE=(SRC=192.168.2.32/27)(DST=SCAN-VIP1)(SRV=DB01_RW)(ACT=accept)))(RULE=(SRC=192.168.2.32/27)(DST=SCAN-VIP2)(SRV=DB01_RW)(ACT=accept)))(RULE=(SRC=192.168.2.32/27)(DST=SCAN-VIP3)(SRV=DB01_RW)(ACT=accept)))

...

cman.ora

DB01_RW=(DESCRIPTION=

(SOURCE_ROUTE=yes)(ADDRESS_LIST=

(ADDRESS=(protocol=tcp)(host=CMAN_HOST)(port=1521))(ADDRESS=(protocol=tcp)(host=SCAN)(port=1521)) <= BLOCKED BY FIREWALL

)(CONNECT_DATA=(SERVICE_NAME=DB01_RW)))


2014 © Trivadis

Network Setup

30


2014 © Trivadis

Conclusion

31

Problems

� Client connections� Oracle Forms 6 / Oracle Client 8� JDBC Thin Drivers� Applications only Server Name/SID/Port as input


2014 © Trivadis


High Availability

32

2014 © Trivadis

Databases on the Clusterware

33

Type Cluster Type License

Single Instance Local DB License

Failover (Cold) Clusters Active-Passive DB License

RAC One Node Active-Passive DB + RAC One Node License

RAC Active-Active DB + RAC License


2014 © Trivadis

High Availability

High Availability for

� Oracle Single Instance Databases

� Oracle Golden Gate

Short RTO, RPO =~ 0

HA Configuration

� Oracle Clusterware (Fail Over Cluster / Cold Cluster)� Server Failures� Instance Failures

� Data Guard (Maximum Availability)� Disaster� Storage Failures� FSFO will be evaluated after the migration

34


2014 © Trivadis

Failover Clusters / Cold Clusters

Active Passive Cluster

Action Script is required with start, stop, check and clean entry points

Trivadis Toolbox has all necessary scripts to build this environment.

35

[+ASM2] crsctl status resource AC.PESS01.DB -pNAME=AC.PESS01.DBTYPE=cluster_resourceACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--ACTION_SCRIPT=/u01/app/oracle/accarda/bin/ac_db.kshACTIVE_PLACEMENT=0CHECK_INTERVAL=30PLACEMENT=restrictedHOSTING_MEMBERS=x101dbadm01 x101dbadm02RESTART_ATTEMPTS=2START_DEPENDENCIES=hard(ora.DATA01.dg,ora.DBFS_DG.dg,ora.RECO99.dg, AC.PESS01.VIP) pullup(ora.DATA01.dg,ora.RECO99.dg, AC.PESS01.VIP)STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.RECO99.dg,ora.DATA01.dg,AC.PESS01.VIP )…


2014 © Trivadis

Failover Clusters vs RAC-One node

� DB License

� TAF for unplanned downtimes

� TAF for planned downtimes

� No Integration with Oracle Tools like dgmgrl, SQLPlus, lsnrctl.

� Needs action script for Database

� Need Action scripts for DBFS, Golden Gate

Failover Clusters

36

� DB License + Separate License

� TAF For unplanned downtimes

� Can two instances be open for max. 30 minutes. All connections will be transferred during this time. No downtime for planned maintenance.

� Full Integration with Oracle Tools like dgmgrl, SQLPlus, lsnrctl.

� Out-of-box product for Database

� Need Action scripts for DBFS, Golden Gate

RAC-One Node


2014 © Trivadis

Database File System

DBFS creates a standard file system stored in database tables.

Files are backed up with Database, synchronized with Data Guard.

Used for Golden Gate Files

37

mount | grep PESS01dbfs-dbfs_user@:/ on /u01/app/oracle/admin/PESS01/dbfs type fuse (rw,nosuid,nodev,max_read=1048576,default_permissions,allow_other,user=oracle)

ls /u01/app/oracle/admin/PESS01/dbfs/ggate/dirprmDESSHUBJ.cpe DESSHUBJ.cps EESSHUBJ.cpb EESSHUBJ.cpe EESSHUBJ.cps desshubx.prmeesshubx.prm jagent.prm mgr.prm resshubj.prm


2014 © Trivadis

Resource Package

38

VIP

Listener

ora.netnnn.network Start: Hard, Pullup

Stop: Hard

Start: Hard, Pullup

ora.<DiskGroup>.dg DatabaseStart: Hard, Pullup

Stop: Hard

Start: Hard, Pullup

Stop: Hard

DBFSStart: Hard, Pullup

Stop: Hard

Golden

Gate

Start: Hard, Pullup

Stop: Hard Start: Hard, Pullup

Stop: Hard

Resource Package with dependencies

VIP: Data Guard log shipment & static connect

Single Instance Oracle Database

DBFS: Golden Gate Trail Files

Golden Gate


2014 © Trivadis

Big Picture

39

x101 (BRU)x201 (HER)

x201db01 x201db02 x201db03 x101db01 x101db02

Prod Prod & Pre-Prod

Env3Env1 Env2

Test/Dev/IntProdProd

Data Guard

Data Guard

FOC

FOC

FOC

FOC

VIP

DB

DBFS

GGATEVIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE

VIP

DB

DBFS

GGATE


2014 © Trivadis

Conclusion

40

Problems

� Linux restriction. Max 15 characters for interface names� Could not create more than 9 VIPs: bondeth0.1004:nn

[root@x201dbadm03 ~]# ifconfig bondeth0.1004: 1bondeth0.1004:1 Link encap:Ethernet HWaddr 90:E2:BA:32:74:48

inet addr: 10.85.44.72 Bcast:10.85.44.255 Mask:255.255.255.0UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

[root@x201dbadm03 ~]# ifconfig bondeth0.1004 :11 10.85.44.70 netmask 255.255.255.0 up

[root@x201dbadm03 ~]# ifconfig bondeth0.1004: 1bondeth0.1004:1 Link encap:Ethernet HWaddr 90:E2:BA:32:74:48

inet addr: 10.85.44.70 Bcast:10.85.44.255 Mask:255.255.255.0UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1


2014 © Trivadis

Requirements

Migrating from

� Linux -> Exadata

� HP-UX -> Exadata

Minimal downtime (2 Hours)

Some VLDB Databases. Up to 4TB

42


2014 © Trivadis

Migrating from Linux -> Exadata

No endian change, both Linux OS

Process:

� Clone Oracle Binaries Linux -> Exadata

� Physical Data Guard (File System -> ASM)

� Perform Switchover

� Remove Data Guard Configuration

� Switch Home (11.2.0.3 PSU Jan14) + Post Upgrade

� Change DB Name, DBID if required

� Delete Cloned Oracle Binaries

43


2014 © Trivadis

Migrating from HP-UX -> Exadata

Endian change

� Logical Migration� Replication Mechanisms (Streams, Golden Gate,..)� Export Import

� Physical Migration� Transportable Tablespaces

Different Components, Different Applications => physical migration wherever possible.

Combining TTS + RMAN Incremental Backups (MOS:1389592.1)

� Explains only how to transport tablespaces. Needs extension for the DB Migration

44


2014 © Trivadis

HP-UX -> Exadata


45

HP-UXEXADATA

ASM FS

� NFS Accessible from both servers

� On Exadata:� Empty database

� Import users

impdp NETWORK_LINK=migration_linkFULL=y include=USER,ROLE,ROLE_GRANT,PROFILE,SYSTEM_GRANT,DEFAULT_ROLE

IMPORT

USERS

ONLINE

NFS

2014 © Trivadis

HP-UX -> Exadata

46

HP-UXEXADATA

ASM FS

� Copy Data Files with RMAN

� Note the SCN of each data file

RMAN> backup as copy datafile '/db/PDOC01/data/CWMLITE04.dbf' tag ... format ' /dbstore01/TVD_Mig/PDOC01/migfiles/CWMLITE_4.tf' ;

NFS

ONLINE


2014 © Trivadis

HP-UX -> Exadata

47

HP-UXEXADATA

ASM FS

� Convert Data Files

� Store in ASM

convert from platform 'HP-UX IA (64-bit)'datafile

'/dbstore01/TVD_Mig/PDOC01/migfiles/CWMLITE_4.tf'format '+DATA01/PDOC01_SITE1/%N_%f.xtf'

parallelism 4;

NFS

ONLINE


2014 © Trivadis

HP-UX -> Exadata

48

HP-UXEXADATA

ASM FS

� RMAN Incremental backup from SCN

set nocfau;backup incremental from scn 50884010185

tag tts_incr_update tablespace 'CWMLITE' format'/dbstore01/TVD_Mig/PDOC01/migfiles/%U';

NFS

ONLINE


2014 © Trivadis

HP-UX -> Exadata

49

HP-UXEXADATA

ASM FS

� Convert Incremental Backups� Special Convert instance (nomount)

- 11.2.0.4- 11.2.0.2 + p12982245 + p14597111

� Apply on the data files

sys.dbms_backup_restore.backupBackupPiece(fname => .../xtts_incr_backup',pltfrmfr => 4 , ...);

NFS

ONLINE

sys.dbms_backup_restore.applyDatafileTo(dfnumber => 33, toname => '+DATA01/pdoc01_site1/drsys_33.xtf‚ ....);

sys.dbms_backup_restore.restoreSetPiece(handle => '/dbstore01/TVD_Mig/PDOC01/migfiles/xtts_incr_backup',...) ;

sys.dbms_backup_restore.restoreBackupPiece(...);


2014 © Trivadis

HP-UX -> Exadata

50

HP-UXEXADATA

ASM FS

� Set tablespaces read only

� Incremental backup

� Apply

Read Only

NFS


2014 © Trivadis

HP-UX -> Exadata

51

HP-UXEXADATA

ASM FS

� Transportable tablespaces to copy metadata

� Tablespaces are belong to the database

impdp network_link=MIGRATION_LINK transport_full_check=no \transport_tablespaces=ARISBPDATA,ARISBPINDEX,ARISDATA \transport_datafiles='+DATA01/paris01_site1/arisbpdata_5.xtf','+DATA01/paris01_site1/arisbpindex_6.xtf','+DATA01/paris01_site1/arisdata_7.xtf'

TTS

Read Only

NFS


2014 © Trivadis

HP-UX -> Exadata

52

HP-UXEXADATA

ASM FS

� Tablespaces read write

� Import metadata

impdp full=y CONTENT=METADATA_ONLY EXCLUDE=USER,PROFILE,... network_link=MIGRATION_LINK

Full MD import

Read Only

NFS

ONLINE


2014 © Trivadis

Conclusion

53

Migration from HP-UX to Linux with 15-30 min. of downtime.

Problems

� No standardized source environment

� Oracle Text => Synchronize� Bad Performance (hang) At TTS Export of Oracle Text (domain) Indexes (Doc

ID 1302676.1)

� Fast Refresh Materialized Views => Rebuild� TTS Does NOT Guarantee an MVIEW Fast Refresh Will Succeed after TTS

Import -- ORA-12034 Possible (Doc ID 1498846.1)

� OLAP analytic workspaces => Migrate manually� Migrating OLAP From 32 Bits to 64 Bits or Across Platforms (Doc ID

352306.1)


2014 © Trivadis

Questions and answers ...

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Umut Bozkurt

Senior Consultant

[email protected]


an exadata database machine setup version doag

Documents