cesnet: data storage · infrastructure overview 15th terena tf-storage task force meeting 22...
Post on 27-Jul-2020
2 Views
Preview:
TRANSCRIPT
CESNET: Data Storage
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
Jan Horníčekjan.hornicek@cesnet.cz
Infrastructure overview
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● 3 geographically separate sites (Pilsen – 8.727 PB, Jihlava – 7.245 PB and Brno – 5.787 PB)
● Based on HSM storage (SATA disks, MAID, tapes)
Praha
Liberec
Pardubice
Brno
Olomouc
Ostrava
Opava
NIX
Internet
AMS-IX
SANETACONET
PIONIER
Jihlava
Děčín
PlzeňKarviná
Zlín
České Budějovice
HradecKrálové
GÉANT
Infrastructure overview
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● 4 different name-spaces
● Backup, archive space accessible by classical file transfer protocols (FTP, SFTP, FTPS, SCP, NFSv4)
● FileSender (5 TB, backed up to tapes)
● dCache (40 TB, backup based on policies)
● ownCloud (40 TB, backed up to tapes)
Pilsen site
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● 8.727 PB raw capacity
● 2 077 users
● 58 VOs
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
Dat
a [T
iB]
Time
Data stored on tapes/MAIDCESNET Pilsen storage
Maximum CapacityTapes usageMAID usage
Tapes+MAID usage
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
500
1000
1500
2000
2500
traf
fic [M
bps]
Time
Data trafficCESNET Pilsen traffic
inout
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
FileSender
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
500
1000
1500
2000
2500
3000
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Dat
a [G
iB]
Time
CESNET filesender used space
Used space
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
10
20
30
40
50
60
traf
fic [M
bps]
Time
Data trafficCESNET Pilsen filender traffic
inout
dCache
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
100
200
300
400
500
600
700
800
traf
fic [M
bps]
Time
Data trafficCESNET Dcache traffic
inout
ownCloud
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
200
400
600
800
1000
1200
1400
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Num
ber
of u
sers
Time
CESNET ownCloud users
Number of users
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
1M
2M
3M
4M
5M
6M
7M
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Num
ber
of fi
les
Time
CESNET ownCloud files stored
Number of files
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
2k
2k
3k
4k
4k
4k
5k
5k
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Num
ber
of fi
les
Time
CESNET ownCloud files per user
Used space
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0
1
2
3
4
5
6
7
8
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Dat
a [T
iB]
Time
CESNET ownCloud used space
Used space
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
6.0
2014/03 2014/04 2014/05 2014/06 2014/07 2014/08 2014/09 2014/10
Dat
a [G
iB]
Time
CESNET ownCloud data per user
Used space
ownCloud - speed up
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● Connecting to database – time consuming
● Reduce overloading of our db node
● Increase overall responsiveness (connection pooling - reuses existing connections with every incoming query)
● Infrastructure for future load-balancing / replication setup
pgpool
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● Acts as a proxy between owncloud (apache) and postgreSQL DB
● Reuses existing connections to DB
● Queues queries to db when no free slot
● Failover ability
● Replication ability (replicate queries over multiple DB servers)
● Load balancing over multiple replicated DB
Thank youfor your attention!
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
pgpool
15th TERENA TF-Storage Task Force Meeting22 September 2014, Uppsala, Sweden
● Connection pooling (24 pools, 2 connections per each)
● Connection limiting -- limit is the size of pool = 48 connections to db
● Technical documentation is being summarized in 'OwnCloud Black Book'
top related