UniCredit Global Backup Infrastructure

Mirco Lissandrini, Team Leader GCC Open Storage

email: Mirco.Lissandrini@unicredit.eu

Milan, 28 May 2013

2

AGENDA

Note:

UniCredit – at a glance

UniCredit Business Integrated Solutions: Identity Card

Global Backup Area Network Infrastructure

The Key Elements of Global Backup Area Network

Facts and Figures of Global Backup Area Network

Local / Remote Cluster DG Configuration and Replication

Backup and Replication Data Flow

Policy Distribution By Type

Case Study: Catalog Disk Layout

Case Study: Synthetic vs. Optimized Synthetic

Questions & Answers

3

Employees: more than 156,000*

Branches: 9,322*

Banking operations

in 22 countries

International network spanning:

~ 50 countries

Global player in asset management: € 157.9 bn

in managed assets*

Market leader in Central end Eastern

Europe leveraging on the region's

structural strengths

* Source: UniCredit Company Profile, data as at December 31, 2012

UniCredit – at a glance

AUSTRIA

ITALY ROMANIA

POLAND

HUNGARY

UK

SLOVAKIA

GERMANY ~ 11.000 Fte’s

( y/y pro forma)

■ 11 Countries*

■ 4 Legal Entities

* UniCredit Business Integrated Solutions operates also in 2 branches, one located in New York and one in Singapore.

UniCredit Business Integrated Solutions is the first concrete milestone within the Group Strategic Plan

2012 announced in November 2011 to be achieved.

Owned by UniCredit, is created from the integration and consolidation of 16 Group companies (among them

UGIS, UCBP, URE, UC) and is dedicated to providing services in the sectors of Information and

Communication Technology (ICT), Back Office and Middle Office, Real Estate, Security and Procurement.

A new business model, unique in the European banking sector, focused on Business needs

(i.e. Commercial Banking, Global Markets, CEE), not only on providing services.

CZ REP.

~ 2,6 mld Revenues by 2012

■ ~ 550 mln IT Investments

UniCredit Business Integrated Solutions: Identity Card

5

Global Backup Area Network Infrastructure

Unified Infrastructure where all resources are shared from 4 Main Data Centers (Masters,

Media Servers, Libraries, Deduplication Appliances, Policies, Clients, Retentions,

Configurations, SLPs, etc.)

Note:

6

The Key Elements of Global Backup Area Network

Master Servers: 4 HP DL580 G7 - 2 local MetroClusters (Italy & Germany) joined into a

GeoCluster with Volume Replication of Shared Disk Groups between sites. Current

version is 7.5.0.5 on Windows 2008 R2 SP1

Media Servers: 24 HP DL380 G8 spread across 4 Data Centers. Current version is

7.5.0.5 on Windows 2008 R2 SP1 and RHEL 6.1

NBU Catalog: 900 GB Catalog Images compressed (2300 GB flat)

Global Data Protected: 22.3 PB overall (18.1 PB on DataDomain vs. 4.2 on Tape –

82% vs 18%)

Tape Libraries: 4 Oracle SL8500 with 48 T10K model A and B drives, around 8200

cartridges whose 4900 are scratch

Deduplication Appliances: 18 Data Domain Appliances installed (DD880, DD890 and

DD990), 4 PB of total space, 18.1 PB of total pre-compression, 2.0 PB of total post-

compression, 9.1x (89.1%) global deduplication ratio and 15.8x (93.7%) best ratio

NetStorage: every Master, Media Server and Data Domain Appliance is connected to

private, not-firewalled and Nexus 5000/7000 based 10GbE network. NetBackup client

link is still a work in progress

Note:

7

Facts and Figures of Global Backup Area Network

Monthly Jobs: 690.000 (22000÷26000 daily)

Monthly Restores: 4000 (mostly DB cloning)

Monthly Successful Rate: 99,6% ÷ 99,7%

Monthly Data Backed Up: 4.5 PB of primary copies (120÷190 TB daily)

Number of NB Clients: 6532 (not unique)

Number of Policies: 1612

Number of SLP: 161

Retention Levels: weekly (1,2 and 3)

montly (1,2,3,6,9 and 18)

yearly (1,2,3,5,7,10 and 11)

Data Distribution: 67% ÷ 70% of backup data are addresses to Deduplication

Appliances, the remaining to Tape Libraries

Storage Unit Reference: all backups with retention less 18 months are addresses to

Deduplication Appliances

DBs vs. FileSystems: 60% ÷ 65% of daily backup data belong to Database systems

(DB2, Oracle, SAP, SQL, Sybase and Teradata)

Note:

8

Local / Remote Cluster DG Configuration and Replication

2 Distinct Clusters

(Local and Remote)

with Global Cluster

Option enabled

Each volume is

mirrored locally and

remotely by Storage

Foundation to get

full-redundancy

Disk Groups are

asyncronously

replicated from local

to remote sites for

Disaster Recovery

reasons

Target Scenario is to

enable Active Master

entity rotation from

site to site

Note:

9

Backup and Replication Data Flows

Every Backup is

duplicated on a

secondary site

indipendently from

the destination

unit

First copy is

written locally

(Tape or Disk),

then it is

duplicated on the

next metropolitan

Data Center

Some strategic

backups require to

be replicated on a

cross-country site

for legal or critical

demands

Note:

10

Policy Distribution By Type

Most of the Policies are

of FileSystem type

(54,2%+36,7%)

DB policies are globally

less than 3,7% (DB2,

Oracle, SAP, SQL and

Teradata)

FileSystem backups are

usually Master initiated,

DBs instead are client

initiated

Policies are separated by

Accounting Code,

Backup Type, Window

and Retention

Note:

DB2 ; 0,25%

FlashBackup-Windows ; 0,75%

MS-Exchange-Server ; 0,57%

MS-SQL-Server ; 0,19%

MS-Windows ; 36,75%

NBU-Catalog ; 0,06%

NDMP ; 0,57%

Oracle ; 1,32%

SAP ; 0,13%

Standard ; 54,21%

Teradata ; 1,70%

Vault ; 0,13% VMware ; 3,39%

Policy Distribution by Type

11

Case Study: Catalog Disk Layout - 1

Master Server suffered of a severe queuing issue in the scheduling rush hours

A lot of tuning changes on EMM, NBRB, NBPEM and NBDB were applied to reduce the

impact of queuing issue

The problem has gotten worse after NetBackup 7.5 upgrade (Catalog info moved into NBDB)

The queuing was solved splitting NBDB and Catalog flat files into two distinct volumes

Note:

NBUlog

G:\ - BS 64K F:\ - BlkSize 64K

NBUCatalog NBDB

Volume Group: USNBUCatalog Volume Group: USNBULog

E:\ - BS 64K F:\ - BS 4K

NBDB NBUCatalog NBUlog

Volume Group: USNBULog Volume Group: USNBUCatalog

SPLITTED UNCHANGED

G:\ - BS 64K

2500 GB 500 GB

500 GB

200 GB

1300 GB

12

Case Study: Catalog Disk Layout - 2

Some other tuning had been applied to NetBackup Relational Database server.conf

-m truncates the transaction log when a checkpoint is done.

-gn 40 number of requests the database server can handle at one time.

-cl 2G minimum cache size, as a limit to automatic cache resizing.

-ch 32G maximum cache size, as a limit to automatic cache growth.

Some EEBs had been installed to improve DB queries from bpdbm and nbpem (i.e. 3084976)

Note:

13

Case Study: Synthetic vs. Optimized Synthetic - 1

NetBackup Synthetic Backup is a feature where a full backup can be synthesized on Media Server

by making use of previous full and subsequent incremental backups reading those component

images and writing a new image.

Because processing takes place on Master and Media Servers instead of the client, synthetic

backups help to reduce the network traffic eliminating or reducing frequency of traditional full

backups

The policy type must be either Standard or MS-Windows.

The Collect True Image Restore Information With Move Detection option must be selected on

the Policy Attributes tab.

The schedule that is created for a synthetic backup must have Synthetic Backup selected.

Note:

14

Case Study: Synthetic vs. Optimized Synthetic - 2

NetBackup Optimized Synthetic Backup is a feature where a full backup can be synthesized on

Storage Server by making use of previous full and subsequent incremental backups without reading

and writing those component images.

Virtual Synthetics use DataDomain’s metadata for synthesizing a full backup;

The big benefit is no data is moved back onto the NetBackup Media Server, therefore an increased

performance will be raised as well as an important server load reduction

Policy requirements are the same of NetBackup Synthetic Backup

DDOS 5.2.x is required and feature must be activated with command:

“ddboost option set virtual-synthetic enabled”

OptimizedImage flag must be present for StorageServer and DiskPool (nbdevconfig –changests and

nbdevconfig –changedp with -setattribute OptimizedImage)

Note:

1. Sunday – Traditional full backup

2. Monday – Saturday – Incremental backups

3. Sunday – Incremental, then Virtual Full

• Virtual Full = regions from traditional full and Incrementals

1 F

2 I I I I I I

3 I VF

NetBackup Server

Data Domain System

DD Boost

15

Case Study: Synthetic vs. Optimized Synthetic - 3

In Synthetic Backup:

- Full-image traverses from the storage (disk or tape) to the Media Server via whatever connection.

- Incremental-image(s) also traverses from the storage to the Media Server.

- Media Server uses its resources (processor, memory, network, storage I/O, etc) to combine

previous full and incremental(s) to create a new "full" backup.

- Media Server then sends this new synthesized image back to the storage via whatever connection.

In Optimized Synthetic Backup:

- Media Server 'instructs' the Storage Server (via the OST API) to synthesize a new image.

- Without sending anything to the Media Server, the Storage Server synthesizes the new image out

of existing images. This is all done inside the storage itself. The Media Server resources are not used.

- When the new synthetic image is successfully created, the Storage Server 'informs' (via the OST

API) the Media Server mission accomplished.

Note:

16

Questions & Answers

Note: