Storage Workshop Summary

Wahid Bhimji University Of Edinburgh

On behalf all of the participants…

What was it?

• Monday / Tuesday this week.• Focused on Tier 2 Grid Storage in the UK.• Good attendance from UK storage

community:– All our SRM technologies dCache/DPM/ Storm – Expertise in the demanding customers - CMS/

ATLAS – but not forgetting the others

Stated Aims

• Data at LHC– Tier 2 storage facing significant challenges.

• Workshop - take a step back and: 1.Identify significant issues from site experiences. 2. Provide training and spread good practice for

current SRM/Storage technologies. 3.Consider other/emerging technologies:– Do these actually address the challenges – If so – what do we need to do to test/develop them.

Overview

Sessions on: • Site Experiences – bottlenecks• SRMs – experiences/tutorials/plans/ comparisons• Hardware• Filesystems and File Access• “The Future”

Some highlights here – see upcoming write-up for full info…

Site Experiences

• Sites (big and small) running well• Dealing with all kinds of

operational issuesIssues:• Upgrade schedule – can gridpp

manage process, to getting features and stability

• Thin(ner) SRM layers - do all ‐ use cases need the full SRM spec - longer term discussion‐

SRM – Storage Resource Manager

Variety of SRM implementations (3 in UK Tier2s) • All increasingly mature – no losers dropping out• Do have different strengths

SRMs

dCache:•Getting easier to use •Well liked by current sites•Certain key valuable features such as replication

DPM:•Most used in UK ( and world)•Support from CERN improved• Development of some features desirable (replication, draining , checking) we can push /help them

StoRM •“Thin” SRM - exploits filesystem•Now mature and more widely used•Questions over support and feature implementation •Needs increasing UK expertise

SRMs

• Converging (in some ways)– eg NFS4, file://• But is there a point in making DPM more like

dCache (in features) or Storm (in Lustre support) …

HardwareSSDs: • Ongoing Studies at

Glasgow• Performance for sure

but is it needed if LHC data structures and access improve

Networking: • 10 Gig for those who

can afford it… channel bonding for others

Procurement Discussions

• Advantages of teaming together with “others” (inc university or T2 or T1) for hardware purchases to get bulk rates.

• Even sharing purchase information (where allowed) is useful.

Filesystems – File Access

File access:• Changes in LHC file access

have big impacts on site storage efficiencies, tuning etc.

• Need to continue studies and keep in close contact with developments

Filesystems

• Other advanced filesystems - GPFS, HDFS can offer features that can be exploited for HEP storage

• As can models of analysis such as Hadoop Map-Reduce

“The Future”

• Brainstorming session covered both short term needs and longer term strategy

• Compile to proper output -> workplan but, to give some examples…

Discussions

• Local file access• User space

expectations• Upgrade

schedule getting the features and stability

•Super lightweight SRMs•Current solutions made more easily deployable and configurable

•Support: Recipes •gridpp certified?

•Coordinated testing of implementations •Regular Workshops!

Emerging Issues / QuestionsEmerging Issues / Questions

Dream storage solutions

Dream storage solutions

Actions for GRIPP STORAGE

Actions for GRIPP STORAGE

Conclusions

• Valuable workshop – In and outside sessions

• Indications of challenges ahead –And work to be done

Conclusions

People say “storage is hard”– So we are lucky to have the GridPP Storage

Community– We need to maintain fora for it.

Some Issues• Local file access – random file patterns cause load• Priorities for Evaluation – Hardware / SRM-Filesystems• Which SRM is best for a Tier 2- not just a big=dCache small=

DPM issue.• Do I buy SSDs? 10 Gig E? More reliable disks? • User space expectations- “I have 2 TB in my pocket why do

you give me 1 on “the grid”• Should GridPP package/certify its own releases – ensure

they are stable + get early access to features• Do we need (to use) a lighter way to access some things on

a grid – at Tier 3s etc. • Can we get more value through shared procurements