redundant datacenter configuration with · pdf fileqnet mpls primary primary 3g private apn...

Real Time Data Processing System

Real Time Data Acquisi7on System

Informa7on systems

Data Sources

Event informa7on ShakeMap informa7on

Event processing ShakeMap processing

Internet

Carrier O&M

BB Station

qnet

MPLS

primary

primary

3GPrivate APN

PrimaryData Center

BackupData Center

Data import & processing

en0

Data export

en0

CE

PE

CE

PE

NTP

SMS

Data export

en0


en0

CE CE CE

SMS

NTP

BB Station

qnet

BB Station

qnet

BB Station

qnet

BB StationLAN

BB StationLAN

BB StationLAN Station LAN

SM Station

SM Station

SM Station

SM Station

SM Station

LAN switch

3G 3G

PEPE

PE

PE Internet

LAN switch

CE

Redundant Datacenter Configurat ion With Antelope

Stefan Radman Open Systems & Services Kinemetrics, Inc www.kmioss.com

Antelope User Group Mee5ng Baku, Azerbaijan

May 2014

Outline

2

•  System Layout

•  Network Failover

•  Applica9on Failover

System Layout

3

Internet

Carrier O&M

BB Station

qnet

MPLS

primary

primary

3GPrivate APN

PrimaryData Center

BackupData Center


en0

Data export

en0

CE

PE

CE

PE

NTP

SMS

Data export

en0


en0

CE CE CE

SMS

NTP

BB Station

qnet

BB Station

qnet

BB Station

qnet

BB StationLAN

BB StationLAN


SM Station

SM Station

SM Station

SM Station

SM Station

LAN switch

3G 3G

PEPE

PE

PE Internet

LAN switch

CE

Network failover

4

Internet

Carrier O&M

BB Station

qnet

MPLS

primary

primary

3GPrivate APN

PrimaryData Center

BackupData Center


en0

Data export

en0

CE

PE

CE

PE

NTP

SMS

Data export

en0


en0

CE CE CE

SMS

NTP

BB Station

qnet

BB Station

qnet

BB Station

qnet

BB StationLAN

BB StationLAN


SM Station

SM Station

SM Station

SM Station

SM Station

LAN switch

3G 3G

PEPE

PE

PE Internet

LAN switch

CE

Dynamic Re-‐rou7ng

Applica9on Failover: System en99es

Real Time Data Processing System

Real Time Data Acquisi7on System

Informa7on systems

Data Sources

Event informa7on ShakeMap informa7on

Event processing ShakeMap processing

5

Applica9on Failover: DataFlow -‐ MS Main System

6

orbserver:bulletin

IRISGSN

ORB ORBrtserve.iris.washington.edu

:18000SeedLink

USGScatalog

M≥8

http://earthquake.usgs.gov

M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb orb2orb slink2orb slink2orb USGS2orb USGS2orb

orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

WebEmailSMS

Data and information flow at Primary Data Center

Data import and processing

Cooperating networks & institutions

Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions

orbserver orbserver ringserver

Data acquisition

ORB

ORB SeedLink

orbserver:acq


Data export


orbserver:exp ringserver

Data export

orb2orborb2orb slink2orb

orb2genc

orb2ringserver

Primary Data CenterPublic IP

orbserver:bulletin

IRISGSN


:18000SeedLink

USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

WebEmailSMS




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions



7


8

orbserver:bulletin

IRISGSN


:18000SeedLink

USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

WebEmailSMS




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions



9

orbserver:bulletin

IRISGSN


:18000SeedLink

USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

WebEmailSMS




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions


Applica9on Failover: DataFlow -‐ BUS Backup System (BUS)

10

WebEmailSMS

orbserver:bulletin

IRISGSN

rtserve.iris.washington.edu:18000

USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

Data and information flow at Backup Data Center



Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions

fromHQ

Analysts

Backup Data CenterPublic IP

Data acquisition

orbserver:acq


Data export


orbserver:exp ringserver

Data Export

orb2orborb2orb slink2orb

orb2genc

orb2ringserver

Applica9on Failover Backup System (BUS)

11

WebEmailSMS

orbserver:bulletin

IRISGSN


USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions

fromHQ

Analysts

•  Concurrent data acquisi7on

•  Processing at both data centers

•  Two produc7on databases

•  Unique IDs

•  las7d replica7on

•  One ac7ve informa7on system


12

WebEmailSMS

orbserver:bulletin

IRISGSN


USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions

fromHQ

Analysts

•  ID Server (dbids)

•  ID Replica7on

(dbids.lastid)

•  Monitoring

(idserver_connect)

•  Failover 7me

•  Authority


13

WebEmailSMS

orbserver:bulletin

IRISGSN


USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions

fromHQ

Analysts

•  Ini7ated by BUS

•  Synchronize local dbids.las7d

•  Ac7vate dbids

•  Change db descriptor to use

local dbids

•  Restart database writers

•  Ac7vate informa7on systems

•  No7fy operators (Email, SMS)


14

•  BUS providing database IDs

•  BUS has authorita7ve database

•  BUS feeding informa7on to the

world

•  BUS prevents ac7va7on of

dbids on MS (armed) orbserver:bulletin

IRISGSN


:18000SeedLink

USGScatalog

M≥8


M≥4.5 archiving

M≥8

qed

RB:9500

RB:9500

RB:9500

RB:9500

RB:9500

SOH

SOH

SOH

SOH orb2orb

orb2orb

orb2orb

orb2orb

orb2orb

orb2orb


orb2dbt

orbdetect

orbassoc

orbevproc

orb2wf

orb2dbt

shake_mon

orbserver:acq

q330

q330

q330

q330

q3302orb

logs

orb2logs

orbdlalert

orbeqalert

dn dnids

orbserver:import

dbids

Officials

AnalystsEmailSMS

ShakeMap

dbloc2

orbeqalert

WebEmailSMS




Broa

dban

d st

atio

nsSt

rong

mot

ion

stat

ions


Applica9on Failover

15

MS •  Check db consistency •  Start rtexec (if not running) •  Verify data flow

•  Check db consistency vs dbids.las7d •  Start dbids

•  Start informa7on systems

BUS

•  Stop db writers (orb2wf & orb2dbt) •  Stop dbids •  Replicate dbids.las7d to MS •  Change db descriptor •  Stop monitoring MS

•  Start db writers •  Start Monitoring MS •  Stop informa7on systems

BUS •  Extract data to external disk (e.g. USB)

trexcerpt –v –e db/db <external disk> <star/me> <end1me> MS •  Stop database writers •  Merge data from external disk

trexcerpt –v –a –e <external disk> <star/me> <end1me> •  Start database writers

Applica9on Failover

16

Important •  Wri_en procedures for stressful situa7ons •  Drills and dry-‐runs for applica7on and network •  At least once a year •  Revise procedures if needed

Applica9on Failover

17

Redundant Data Center Configura9on

18

THANK YOU

redundant datacenter configuration with · pdf fileqnet mpls primary primary 3g private apn...

Documents