conflicts and synergies between industry and academiaevent.twgrid.org/isgc2008/presentation...
Post on 30-Sep-2020
6 Views
Preview:
TRANSCRIPT
© 2006 Open Grid Forum
Conflicts and Synergiesbetween
Industry and AcademiaCraig A. Lee, President, OGF
International Symposium on Grid ComputingAcademia Sinica, Taipei, Taiwan -- April 9, 2008
2© 2008 Open Grid Forum
Conflicts -- Different Goals
• Academia:• Computer Scientists want to
"design the world" -- define the entire framework
• Discovery, workflow, security• Technical Transparency• Publications and grants• Explore fundamental issues,
properties, and design space for a given topic area
• Industry:• Industry wants to build just
what they need for immediate goals
• Cluster scheduling• Competitive advantage• Short time to market• Identify specific capabilities
-- points in the design space -- around which to build products and markets
3© 2008 Open Grid Forum
Innovation “no man’s
land”OGF
Driving Innovation
Research push Market pull
Phase 1Solution proposal
Phase 2Prototype
Phase 3Pre-commercial product/service
Phase 4Commercial
product/service
Phase 0Research
Market Pull
Managing the Technology Maturation Process
Technology Readiness Levels
Ulf Ulf DahlstenDahlsten, Director , Director ‘‘EmergingEmergingTechnologies and InfrastructuresTechnologies and Infrastructures’’
4© 2008 Open Grid Forum
The Goal: Synergy!
How to manage “technology maturation”?How to:
1. Facilitate the development of exciting research directions and the desire for thorough technical understanding, while
2. Coordinating the wide-scale application and adoption of specific capabilities that provide value to IT organizations and build markets?
Where are we now?• What have we accomplished?
Where are OGF and the grid community going?• What does the technical landscape look like and
where are the opportunities for synergy?
5© 2008 Open Grid Forum
What Have We Accomplished?
• Many, many operational grids comprised of hundreds of thousands of nodes
• Brand Identity• Strong world-wide community• Growing adoption of standards
• DRMAA, HPC Basic Profile, JSDL, BES• Others in pipeline
• Vibrant research topics• Web 2.0, Virtualization, Cloud Computing,
Green IT
6© 2008 Open Grid Forum
An Example: EGEE
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
Bob Jones, EGEE Project Director
7© 2008 Open Grid Forum
European Grid Initiative
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
8© 2008 Open Grid Forum
Important Activities at OGF
• HPC-Basic Profile• JSDL & BES for HPC job submission• Interoperability layer for cluster schedulers• Demos at SC06 and SC07 with MS, HP, IBM, Platform…
• GLUE: Resource Information Model• On-going work w/ DMTF on harmonizing GLUE and CIM
• OGSA-WG: Open Grid Services Architecture• Specifications developed & now driving adoption
• GridRPC-WG: Grid Programming Model• Recently progressed to full recommendation
• SAGA-WG: Simple API for Grid Applications• Reference Model WG
• Infrastructure Management -- Data Centers8
9© 2008 Open Grid Forum
OGSA Spec AdoptionProject/Spec
WS-
Nam
ing
RN
S
OG
SA-B
ES
OG
SA-B
yteI
O
WS-
DA
I
OG
SA-B
SP 2
.0
JSD
LW
SRF
Bas
ic
Prof
ileH
PC B
asic
Pr
ofile
Globus ? yes yes yesUnicore 6 no yes yes yesFujitsu USMT will yes yes yes yesMicrosoft CCS yes yes yesGenesis II yes yes yes yes no will yes yes yesMarty yes yes yesOMII-UK yes yes yes yesGridSAM yes yes yesCrown yes yes yesPlatform yes yes yesOGSA-DAI yes yesNAREGI No May Yes May Yes yesOsamu yes
10© 2008 Open Grid Forum
OGF Reference Model:a Grid Component Instantiation Dependency Graph
OGF Reference Model, v2.0E Draft, D. OGF Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay
11© 2008 Open Grid Forum
Grid Management Entities
OGF Reference Model, v2.0E Draft, D. OGF Reference Model, v2.0E Draft, D. SnellingSnelling, Fujitsu, P. Strong, eBay, Fujitsu, P. Strong, eBay
12© 2008 Open Grid Forum
Some OGF-22 Workshops
• Financial Services• Hedging, Risk Mgmt, Protocol pricing• Wall Street firm just approved $100M for next generation grid• Scaling, storage, power, cooling, app development
• Cloud Computing• Keynote by Irving Wladawsky-Berger of IBM and MIT• Workshop room packed• Cloud Interoperability, Reliability, Security
• Data• Digital Libraries, Digital Repositories -- data explosion• Metadata Exchange key topic• Potential collaboration w/ SNIA
• Spec Adoption Track• GIN, SAGA, DRMAA, HPC-BP• Managed efforts for key stakeholders and potential adopters
13© 2008 Open Grid Forum 13
OGF-Europe Funded by EU!
• The EU recognises interoperable software standardsinteroperable software standards are important for broader, more mainstream adoption
• OGF-EUROPE crucial for realization of past 8 years of European investments in Grid computing into actionable and sustainable results, targeting the Small-to-Medium Enterprise (SME) sector
• EC Funding: Service & Software Architectures, Infrastructures & Eng.
• FP6, 2002‐2006, €€78 Million78 Million
• FP7, 2007‐2013, €€120 Million120 Million
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
14 14
Some specifics …
5 Technical Expert Fellowships5 Student Scholars2 Surveys on trends & grid practices 3 Reports supporting OGF Tech. Strategy & Roadmap 4 Grid adoption challenge reports 6 Community best practice reports 4 Workshops on standardization challenges6 Community outreach seminars
Including 4 in-depth tutorials
2 OGF-Europe international events
Dave Wallom, OERC
First International Event• OGF‐23, June 2‐6, 2008, Barcelona
– Hosted by Barcelona Supercomputing Center– 500 + participants: EU projects, industry leaders, standards developers and end users
• Community Outreach Seminars– Digital Repositories– Geospatial, GENESI‐DR
• Co‐located with BEinGRID Industry Days– EU project with 23 experiments with 75 partners in applied research and industry
– Financial modeling, retail mgmt, design and simulation, earth observation, groundwater modeling, fluid dynamics, and seismic processing
Who’s in OGF‐Europe ?
QuickTime?and aFF (LZW) decompreseeded to see this pict
Dave Wallom, OERC
Contact info@ogfeurope.eu to learn more
17© 2008 Open Grid Forum
The OGCThe OGC--OGF CollaborationOGF Collaboration
• Promote best practices and international standardization for distributed geospatial data processing capabilities that is:•• TransparentTransparent -- users is not aware of the infrastructure•• InteroperableInteroperable -- the resources work together•• ScalableScalable -- small local, to massive distributed platforms
18© 2008 Open Grid Forum
Some Geospatial Factoids
• Open Geospatial Consortium• www.opengeospatial.org
• “Helping the World to Communicate Geographically”• Any type of geospatial data• Anything that goes on a map
• A Few Current OGC Standards• Web Map Server (WMS)• Web Feature Server (WFS)• Web Coverage Server (WCS)• Catalog Service for Web (CSW)
• Half of all Web 2.0 URLs registered on programmableweb.com are geospatially related
• Google pushing KML through OGC process
19© 2008 Open Grid Forum
Immense Applicability of Geo Data
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
Image: E. Gennai, C. Terbough, ESRI
Estimates vary, but ~80Estimates vary, but ~80--90% of all data collected or 90% of all data collected or produced by the human race is geospatially referencedproduced by the human race is geospatially referenced
20© 2008 Open Grid Forum
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
Carl Reed, OGC CTO:Carl Reed, OGC CTO:““GRID Technology must GRID Technology must be an integral component be an integral component of the of the GeoWebGeoWeb!!!!””
21© 2008 Open Grid Forum
State of the Collaboration• Initial Goals
• Integrate OGC's Web Processing Service (WPS) with a range of"back-end" processing environments• Web 2.0, basic web services, local/national grid infrastructures
• Integration of WPS with programming and workflow tools• Swift, SAGA, DAGMan, Pegasus, Kepler, Taverna, and Triana• … just to name a few
• Integration of OGC Federated Catalogues and Data Repositories with grid data movement tools
• Outcome from OGF-22 Collaboration Workshop, Feb. 26, 2008• Workshop report, set-up project web site, email list• Geospatial grid sessions at Euro Geophysical Union in April in Vienna• Coordinate June meetings (June 2-6)
• OGF-23 in Barcelona with OGC-TC in Potsdam• OGF-Europe sessions on Digital Repositories
• Invited paper in November IEEE Computer (just in time for SC08)•• OGF invited to participate in OGC OWSOGF invited to participate in OGC OWS--6 6 testbedtestbed planning processplanning process
22© 2008 Open Grid Forum
What is OWS?• OGC Web Service Testbed• Annual process where sponsors identify specific
demonstration targets• Larger number of participants supply in-kind resources
• OWS-5 recently finished (March, 2008)• OWS-5 Sponsors
• BAE Systems - National Security Solutions• Federal Geographic Data Committee (FGDC/USGS)• Google• Lockheed Martin Integrated Systems & Solutions (lead org)• Northrup Grumman• US National Aeronautic and Space Administration (NASA)• US National Geospatial-Intelligence Agency (NGA)
• ~70 Participants•• 4x4x--5x Return on Investment5x Return on Investment
•• Strength of CollaborationStrength of Collaboration
23© 2008 Open Grid Forum
OWS-5• Five Threads
• Sensor Web Enablement (SWE)• Geo Processing Workflow (GPW)• Information Communities and Semantics (ICS)• Agile Geography• Compliance Testing (CITE)
• Many videos presented at final outbrief• Sensor Web 2.0 Experiments• Data Architecture Views• Conflation Processing• Web Coverage Processing Service
24© 2008 Open Grid Forum
OWS-5 NASASensor Web Demo
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
EO-1 tasked to collect imagery on Northern San Diego Countywildfires, UAV track data integrated in Google Earth
25© 2008 Open Grid Forum
Planning for OWS-6
• “Whiteboard” session• March 28 in St. Louis• All manner of ideas aired
• FAA AIXM (Aeronautical Info Exchange Model)• SOA for GEO• Service Lifecycle• Geo Rights Management• WMS rehosting• Workflow and WPS• Sensor Web Enablement• OGF INFOD for event pub/sub
•• OWSOWS--6 Sponsors Meeting6 Sponsors Meeting•• April 30, 2008, Herndon, Virginia, USAApril 30, 2008, Herndon, Virginia, USA•• WPSWPS--toto--Grid is a key goal on the tableGrid is a key goal on the table
26© 2008 Open Grid Forum
Switching Gears:“To Distribute or Not To Distribute”
• Prof. Satoshi Matsuoka, TITech• Keynote at Mardi Gras Conference, Baton
Rouge, Jan.31, 2008
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
• In the late 90s, petaflops were considered very hard and at least 20 years off …
• After 10 years (around now) petaflops are “real close” but there's still no "global grid”
• What happened?
• … while grids were supposed to happen right away
27© 2008 Open Grid Forum
What Happened?• It was easier to put together massive clusters than to get
people to agree about how to share their resources• For tightly coupled HPC applications, tightly coupled
machines are still necessary• Grids are inherently suited for loosely coupled apps
(e.g., Monte Carlo, Parameter Sweep), or enabling access to machines and data, and the integration of the two
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
• With Gilder's Law, bandwidth to the compute resources will promote thin client approach
• Example: Tsubame machine in Tokyo
Phil Papadopoulos @ SDSC
28© 2008 Open Grid Forum
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
S. Matsuoka, TITech
29© 2008 Open Grid Forum
So How to Use a Large Machine?
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
Question:Question: So whatSo whatdoes this job sizedoes this job sizedistribution distribution meanmean
for the userfor the userdata distribution? data distribution?
Satoshi Matsuoka, TITech
• How to feed a large machine with lots of users w/ lots of decentralized data?
• What about data interoperability?• How to make it easy to use?• How to virtualize it?
30© 2008 Open Grid Forum
Virtualization and Cloud Computing
•• ““ThereThere’’s Grid in them s Grid in them tharthar clouds!clouds!””• I. Foster’s blog, ANL & UC, Jan. 8, 2008
• Clouds have a very simple user API effectively hiding all the complexity of an ad hoc grid on the back-end• e.g., Amazon’s EC2 & S3, IBM’s Blue Cloud
and others …• If so, will this enable mass-market grids?
• Users don’t have to be aware of using “a grid”• If so, what does “cloud interoperability” require?
• Is virtualization a means of achieving this?•• Major opportunity for synergyMajor opportunity for synergy
31© 2008 Open Grid Forum
How to Manage Sets of VMs in Distributed Environments?
• Virtual Workspaces?• Dynamically provisioned environments
• Implementation using VMs• Encapsulated configuration and fine-grained
enforcement• Easy way to build VMs?
• rPath, OSFarm (CERN), Bcfg2 (ANL)• Managing Virtual Clusters
•• ContextualizationContextualization
32© 2008 Open Grid Forum
Deploying Workspaces Remotely
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Workspace-Workspace metadata
-Pointer to the image-Logistics information
-Deployment request-CPU, memory, node count, etc.
VWSService
K. Keahey, ANL, SC07
33© 2008 Open Grid Forum
Interacting with Workspaces
Poolnode
Trusted Computing Base (TCB)
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
The workspace service publishesinformation on each workspace
as standard WSRF ResourceProperties.
Users can query thoseproperties to find out
information about theirworkspace (e.g. what IP
the workspace was bound to)
Users can interact directly with their
workspaces the same way the would with a
physical machine.
VWSService
K. Keahey, ANL, SC07
34© 2008 Open Grid Forum
Opportunities for Synergy• Resolving VM deployment issues
• Do they have public IP addresses and can find each other?• Can they share storage?• Can I integrate them into my Virtual Organization?
• Virtual Cluster and Contextualization• Networking, shared storage, name resolution, etc. that will be
portable across sites and implementations• Put VMs in a deployment context of the Grid, site, and other VMs• Common understanding among the image “vendor”, deployer,
and user• Continuing work with DMTF
• Identify/write use cases to feed DMTF virtualization work, e.g. portals, aggregated virtualized resources, cloud computing, etc,leveraging OGF’s Grid-Virt use case work as appropriate
• Packaging: OVF vs rPath. Evaluate, identify, and input changes to potentially merge OVF and rPath so there is a single packaging format for the next version of the OVF spec
35© 2008 Open Grid Forum
Spaceship Earth
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
36© 2008 Open Grid Forum
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
… and Climate Change
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime?and aTIFF (LZW) decompressor
are needed to see this picture.
With thanks to Carl Reed
37© 2008 Open Grid Forum
Global demand outstripping supply 3:1
Rich media and social computing
New York City
Demand for Data Centers
LondonSource: www.Bastionhost.com
Bill St. Arnaud
38© 2008 Open Grid Forum
Zero Carbon Strategyfor Cyber-infrastructure
1. More efficient computers and network equipment
2. Virtualization of computer and networks and relocated to “zero carbon” data centers
3. Deployment of “zero carbon” data centers at remote renewable energy sites linked by optical networks
4. New network and computer architectures that take advantage of remote zero carbon data centers
Bill St. Arnaud
39© 2008 Open Grid Forum
Example:Grid in a Box at a Windmill
Source: www.Bastionhost.com
Bill St. Arnaud
• Extensive use of virtualizationvirtualization and fiber opticsfiber optics to put the computing where the power is
• Enable the workload to “follow the wind”or
“follow the sun”
40© 2008 Open Grid Forum
Solar Computing Tiles
Satoshi Matsuoka, TITech
If we make some assumptions about performance levels…
41© 2008 Open Grid Forum
How Much Can a Building Compute?
Satoshi Matsuoka, TITech
• Typical House• Assume 500m2 surface area• 20% efficiency• ~ 2 TeraFlops / house
• 200m Skyscraper• Assume 50m x 50m base,
40,000 m2 surface area• 25% efficiency• ~ 1/2 BG/L compute capacity
42© 2008 Open Grid Forum
A Call to Action
• Look for those opportunities for synergy• Nurture the grid research community
• Good events on hot topics• Broad community involvement• Let a thousand flowers bloom (innovation)
• While targeting key efforts that produce high impact results•• Efforts that connect the investment of time, money Efforts that connect the investment of time, money
and people with concrete resultsand people with concrete results• For example, HPC-Basic Profile
• Seek natural adjacencies for the development and use of grid technology
43© 2008 Open Grid Forum
A Summary of Opportunities
• Technical• Data access and integration• Geospatial data and applications• Data centers• Virtualization• Cloud Computing• Green IT, …
• Organizational• OGF-Europe• DMTF, SNIA, OGC• IBM, eBay, MS, Platform, …
44© 2008 Open Grid Forum
A Shameless Plug
• OGF-23 in Barcelona• Barcelo-Sants Hotel• June 2-6, 2008
• OGF-24 in Singapore• Biopolis• September 15-19, 2008
• OGF-25 with EGEE• Somewhere in Europe, Spring 2009
• And a final bit of wisdom …
45© 2008 Open Grid Forum
The Real Reason why there areno Penguins at the North Pole
Thank you. Any Questions?Thank you. Any Questions?
top related