service and support for science it lunchveranstaltung herbst 2014
Post on 14-Dec-2015
215 Views
Preview:
TRANSCRIPT
Service and Support for Science IT
Lunchveranstaltung Herbst 2014
S3IT
Peter Kunszt
PhD in Theoretical Physics (University of Bern)
Postdoc in Astrophysics / Cosmology, Johns Hopkins University Baltimore, USA: Sloan Digital Sky Survey Science Archive (Virtual Observatory), 3Y
CERN IT Department, Geneva, Data Management Section head and Project Manager for EU projects , 5Y
CSCS, Lugano: Build Swiss Tier 2 for CERN, Swiss Grid Initiative, 3Y
ETH Zürich: Head of SyBIT Projects for SystemsX.ch, 5Y
UZH: Heading S3IT
S3IT
Outline
Science is changing
Challenges due to the change
Addressing the challenge
Infrastructure
Organization of S3IT
S3IT
A Digital World
Scientific Discovery driven by new instrumentation
S3IT
Scientific data doubles every year
Changes the nature of scientific computing
Cuts across disciplines …. eScience
It becomes increasingly difficult to extract knowledge
An Exponential World
19701975
19801985
19901995
2000
0.1
1
10
100
1000
CCDs Glass
Slide by Alex Szalay, JHU
S3IT
Not only scientific data!
20% of the world’s servers go into centers by the “Big 5”
– Google, Microsoft, Yahoo, Amazon, eBay
An Exponential World
Slide by Alex Szalay, JHU
S3ITScience is Changing
THOUSAND YEARS AGOscience was empirical describing natural phenomena
LAST FEW HUNDRED YEARStheoretical branch using models, generalizations
LAST FEW DECADESa computational branch simulating complex phenomena
TODAYdata intensive science, synthesizing theory, experiment and computation with statistics ►new way of thinking required!
Slide by Alex Szalay, JHU
S3IT
Change of Culture
Single person discoveries Large Collaborations
S3IT
Change of Culture
Single person discoveries Large Collaborations
Citizen Science
S3IT
Luckily, there’s Moore’s Law
S3IT
Moore’s Law – no mo(o)re?
S3IT
Scientific Data Analysis Today
Data is produced everywhere, never will be at a single location
Data grows as fast as our computing/instrument power
Many labs have their own power-workstation or mini-cluster
Hitting the cooling and power wall
Moore’s law is not as it used to be – more complex solutions
Trouble storing even the produced data stream
Not scalable, not maintainable…
S3IT
Fire and forget...
Often, you do not want to be bothered with computing details
IT JUST NEEDS TO WORK!
S3IT
Widening Complexity Gap
Standard Computing
GAP Research Needs
Desktop computing, storageHelpdesk, supportInternet, Wikipedia, ..
AlgorithmsModels, StatisticsVisualizationsData analysisPublication
Local IT ResourcesCentral IT Services
Research laboratoriesCore Facilities
S3IT
Challenge: Scale Up
High Throughput Instruments
– Much larger data volumes
– Increased data complexity
Large Collaborations
– More people
– More experiments and measurements
– More coverage
BIG
Eve
ryth
ing
S3ITScience IT
Connect IT and Science
Dedicated support forcomputations and dataanalysis
SPEED : faster time to solution
ACCESS to competitive infrastructure
ENABLE : remove barriersnew possibilities
Speed
Access
Enablement
S3IT
Sure, we all believe in miracles...
S3IT
Gray’s Laws of Data Engineering
Jim Gray:
Scientific computing is increasingly about data
Need scale-out solution for analysis
Take the analysis to the data!
Start with “20 queries”
Go from “working to working”
Slide from Alex Szalay, JHU
S3IT
What does that mean?
1. Understand your data = understand your problem
2. Reduce data wherever possible – think about what is worth keeping vs. reproducing
3. Focus on doable chunks of work and questions
4. Build programs and systems that can scale
S3IT
And: No one size fits all – all research is different by nature
S3IT
Providing the ‚Miracle‘: A lot of this is Scientific Work by itself!
Computer Science
Scientific Computing, Research Informatics
Data Science
• Department of Informatics
• Institute for Computational Science
• Domain Informatics – Bioinformatics, Medical Informatics, Geoinformatics, ....
Lots of PhD theses to come!
S3IT
Results of that research needs to be applied!
Software engineering
Code optimization and scaling
Visualization
Applied statistical analysis
Automation of workloads
Data storage and management
Maintenance ! ! !Students don‘t have time for
that and don‘t get any
recognition
S3IT
Like a Formula 1 Racing Team
S3IT
Informatikdienste
Teamwork in Science IT
CoreFacilities
CoreFacilities
DomainInformatics
Research GroupsProjects
ProjectsProjects
Research GroupsResearch GroupsResearch Groups
FacultiesInstitutes
Departments
FacultiesInstitutes
Departements
Department of Informatics
CSCS
externalinternal
VendorsVendors
VendorsIndustrial Partners
Institute for Computer Science
CoreFacilities
CoreFacilities
CoreFacilities
Science IT @ETH, Univ of X
Zentrale Informatik
CoreFacilities
CoreFacilities
Local IT Support
S3IT
Science IT as a Service
BOOTSTRAP: Consultancy• Research context and perspective• Categorization of problem in terms of
Simulation, Data, Processing, Publication• Map to available infrastructure• Plan Support Service as a Project (time, cost, ..)
DELIVERY: Project execution• Setup of infrastructure, software, integration• Automation, analysis, visualization• Training of the users on the workflow• Continuous Support, feedback and iterative improvement
FINISH• Conclusion and publication• Reusability and sustainability measures
S3IT
Categorization of Infrastructure
Computer is the Resesarch Instrument : ‚Supercomputing‘– Simulations of phenomena
– Needs the largest computers you can get
– Theoretical physics, astrophysics, mathematic, computational chemistry, biochemistry, meteorology
– Simulations also generate a lot of data, models.
– Continuous usage.
Our job: Provide access to necessary Infrastructure
– Support
– Maintenance
– Software optimization
– Data storage and handling
S3IT
Categorization of Infrastructure
Computer is a tool, a workhorse– Statistical analysis, parameter studies
– ‚Big Data‘ processing
– Visualization
– Life science, biochemistry, geography, medicine, digital humanities, banking and finance, computer science, ...
– Very heterogeneous requirements
– Non-continuous use
Can be large!
1. Server computing
– Interactive work, person sitting in front of the system
2. Cluster computing
– Automated workloads, many computers at once
S3IT
Categorization of Infrastructure
1. Server computing
– Interactive work, person sitting in front of the system
2. Cluster computing
– Automated workloads, many computers at once
Our job: Provide access to individualized, custom servers and clusters
– Assure scalability
– Keep costs down
– Maintenance, support
– Automation tooling, workflows
– Data management and data processing
– Standardized processes
S3IT
Supercomputing
UZH operates local supercomputing since almost 10 years
• Irchel Datacenter
• Supported and operated centrally
BUT: Getting ‚old‘ – over 4 years
• Not competitive
• Not power-efficient
• Expensive in maintenance
S3IT
The Science Cloud
Scale by re-centralizing individual local infrastructure
– One hardware size fits all
– But individual delivery of clusters and servers!
– Possible due to virtualization
NEW
Physical Hardware
‚Virtual‘SimulatedHardware
S3IT
Schrödinger Supercomputer Local Computing
Infrastructure Today
Maintained by S3IT / ZI
Supported by S3IT / ZI
Standardized Tools
Locally installed and maintained
Own tools and developments
S3IT
UZH Science Cloud
Maintained by S3IT / ZI
Supported by S3IT / ZI
Toolset : both standard and own tools
NEWUZH HPC@CSCS Local Computing
Infrastructure 2015
Maintained by CSCS
Supported by S3IT / ZI
Standardized Toolset
Locally maintained
Own tools and developments
S3IT
UZH HPC@CSCS Local Computing UZH Science Cloud
Infrastructure 2016
Maintained by CSCS
Supported by S3ITLocally maintained
Own tools and developments
Maintained by S3IT
Supported by S3IT
S3IT
Storage Storage Storage
S3IT
Cost of a Petabyte is not the problem
From backblaze.comOctober 2014
S3ITRead Petabytes in short time is!
Current problems are not easy to parallelize and scale!
• 10-30TB ‘easy’
• 100-200TB doable
• 500TB+ very difficult
Moving 100TB over the network (sequentially)
• 1Gbps – 10 days
• 10Gbps – 1 day : but needs a dedicated connection!
• Physically? – FedEx
S3IT
Categorization of Science IT Problems
• Models• Theory
• Analysis• Mgmt
• Scaling• Autom.
• Share• Publish
Simulation Data
ProcessingPublication
S3IT
Managing the Data Lifecycle
Conscious data production, reduction, usage
• Ask the questions you need to ask as quickly as possible
• Delete data wherever possible, keep starting points (freezer?)
• Aim for reproducibility on all levels on first principles – document steps
Automated data processing pipelines
• Scale by automation
• Ability to re-run simulations and data analysis on a push of a button
• Extract and keep all metadata
Publication and archiving
• Publish everything to be easily reproducible by others
• Archive only what you need or what is mandatory
S3IT
Setup of Service and Support for Science IT
Director
Office
Infrastructure Projects Collaborations
HPCCloud
Storage
Life ScienceGeoscience
PhysicsHumanities
...
NationalInternational
Industrial
S3IT
Setup of Service and Support for Science IT
Informatikdienste
Zentrale Informatik
Vice President Law & Economics
S3IT
Organisation
Core Team
Site Team
Site Team
EE
EE
EE
EE
EE
...
...
EE = Embedded ExpertsProject work directly in the groups.
Site TeamsJoining forces with local institute-IT experts. Support and provisioning of access and software on site.
Core TeamConsultancy in core competences, central infrastructure, project management
S3IT
Funding
• University Core Funding for the core team
• Site Teams: Funding through local institutes, faculties or 3rd party projects – service charges
• Embedded Experts: 3rd party projects – co-applicants on project proposals
S3IT
What to expect of S3IT
Science IT as a Service - Consulting
Support through projects
Software, Workflows, Infrastructure is optimized to the needs of the research problem, not the other way round
Our operational concept follows the cloud model to meet the very heterogeneous needs of the UZH research groups, while working with standardized, commodity hardware and software
Scalability, Extensibility, Reusability are our guiding principles
S3IT
Already a success story
• Started with 1 person in January
• 10 people now, 5 on core, 5 on other funds
• 20 project applications written, 10 projects already approved and running, 10 projects in the pipeline, some already pre-approved
• Many new project ideas as a result of consultation
• Projects include
• Life Science (imaging, genomics, proteomics)
• HPC optimization
• Digital humanities, art history, ..
• Industrial cooperations
S3IT
What next?
How long does the data growth continue?
How far can we scale?
What new technologies will make our life easier or harder?
Let‘s find out together
S3IT
Please visit usNext Lunch Events
Generating Hypotheses from Large Data: Lars Malmström, S3IT
Big Data in Art History: Thoms Hänsli, Ditigal Art History UZH
Visual Analytics, understanding interactions: Markus Grau; Business Alliances, Guido Oswald; SAS
Kognitive Systems, how ‚artificial intelligence‘ will revolutionalize research and education – Karin Fey, IBM
Cloud Services at the UZH – ZI
http://www.s3it.uzh.ch/
top related