condor : a concept, a tool and a model
DESCRIPTION
Condor : A Concept, A Tool and A Model. The Condor Project (Established ‘85). Distributed systems research performed by a team that faces software engineering challenges in a UNIX/Linux/NT environment, active interaction with users and collaborators, - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/1.jpg)
Miron LivnyComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected]://www.cs.wisc.edu/~miron
Condor:A Concept, A Tool
and A Model
![Page 2: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/2.jpg)
www.cs.wisc.edu/condor
The Condor Project (Established ‘85)
Distributed systems research performed by a team that faces
software engineering challenges in a UNIX/Linux/NT environment,
active interaction with users and collaborators, and the daily maintenance and support
challenges of a real-life distributed production environment.
Funding - NSF, NASA,DoE, DoD, IBM, INTEL, Microsoft and the UW Graduate School
.
![Page 3: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/3.jpg)
www.cs.wisc.edu/condor
National Grid Efforts
› National Technology Grid - NCSA Alliance (NSF-PACI)
› Information Power Grid (NASA)
› Particle Physics Data Grid (DoE)
![Page 4: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/4.jpg)
www.cs.wisc.edu/condor
Applications
› Optimization - UW, ANL, NW
› High Energy Physics - INFN, UNM, UW, Caltech
› Biological Sciences - UW, UMN
› Animation - UW, C.O.R.E
› Software Engineering - Oracle
› JAVA - NASA
![Page 5: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/5.jpg)
www.cs.wisc.edu/condor
CS Collaborations
› Argonne National Lab (Globus) - Grid middleware
› Universitat Autonoma de Barcelona - Scheduling of Master-Worker Applications
› Clark Atlanta University - User Interfaces
![Page 6: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/6.jpg)
www.cs.wisc.edu/condor
Funding Distribution
Source FTEs 50% RAs Comments
Federal 3.5 12.0
I ndustry 3.0 Unrestricted grants
Grad School 2.0 2.0
128 account 0.5 "For f ee" support
Total 6.0 17.0
![Page 7: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/7.jpg)
www.cs.wisc.edu/condor
Concept(s)
![Page 8: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/8.jpg)
www.cs.wisc.edu/condor
“ … Since the early days of mankind the primary motivation for the establishment of communities has been the idea that by being part of an organized group the capabilities of an individual are improved. The great progress in the area of inter-computer communication led to the development of means by which stand-alone processing sub-systems can be integrated into multi-computer ‘communities’. … “
Miron Livny, “ Study of Load Balancing Algorithms for Decentralized Distributed Processing Systems.”, Ph.D thesis, July 1983.
![Page 9: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/9.jpg)
www.cs.wisc.edu/condor
Every Communityneeds a
Matchmaker!
![Page 10: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/10.jpg)
www.cs.wisc.edu/condor
Why? Because ...
.. someone has to bring together members who have requests for goods and services with members who offer them. Both sides are looking for each other Both sides have constraints Both sides have preferences
![Page 11: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/11.jpg)
www.cs.wisc.edu/condor
High Throughput Computing
For many experimental scientists, scientific progress and quality of research are strongly linked to computing throughput. In other words, they are less concerned about instantaneous computing power. Instead, what matters to them is the amount of computing they can harness over a month or a year --- they measure computing power in units of scenarios per day, wind patterns per week, instructions sets per month, or crystal configurations per year.
![Page 12: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/12.jpg)
www.cs.wisc.edu/condor
Computing is a Commodity
Raw computing power is everywhere - on desk- tops, shelves, and racks. It is cheap dynamic, distributively owned, heterogeneous and evolving.
![Page 13: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/13.jpg)
www.cs.wisc.edu/condor
Master-Worker (MW) computing is common and
Naturally Parallel.It is by no means
Embarrassingly Parallel.
Doing it right is by no means trivial.
![Page 14: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/14.jpg)
www.cs.wisc.edu/condor
A Tool
![Page 15: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/15.jpg)
www.cs.wisc.edu/condor
Our Answer to High Throughput MW Computingon commodity resources
![Page 16: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/16.jpg)
www.cs.wisc.edu/condor
The Condor System A High Throughput Computing system that supports
large dynamic MW applications on large collections of distributively owned resources developed, maintained and supported by the Condor Team at the University of Wisconsin - Madison since ‘86. Originally developed for UNIX workstations. Fully integrated NT version is available. Deployed world-wide by academia and industry. More than 1300 CPUs at the U of Wisconsin. Available at www.cs.wisc.edu/condor.
![Page 17: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/17.jpg)
www.cs.wisc.edu/condor
0
200
400
600
800
1000
1200
1400
'88 '94 '99 '00
Other
CS
Condor CPUs on the UW Campus
![Page 18: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/18.jpg)
www.cs.wisc.edu/condor
Some Numbers-CS PoolTotal since 6/98 4,000,000 hours ~450 years“Real” Users 1,700,000 hours ~260 years
CS-Optimization 610,000 hoursCS-Architecture 350,000 hoursPhysics 245,000 hoursStatistics 80,000 hoursEngine Research Center 38,000 hoursMath 90,000 hoursCivil Engineering 27,000 hoursBusiness 970 hours
“External” Users 165,000 hours ~19 yearsMIT76,000 hoursCornell 38,000 hoursUCSD 38,000 hoursCalTech 18,000 hours
![Page 19: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/19.jpg)
www.cs.wisc.edu/condor
A Model for ...
![Page 20: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/20.jpg)
www.cs.wisc.edu/condor
CS-Domain Collaborations
Multi disciplinary research that advances the state-of-the-art in CS and a domain science. Based on mutual respect and understanding
of objectives, needs, constraints and culture Leverage expertise, resources and funding Enables experimental Computer Science Enables speculative science
![Page 21: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/21.jpg)
www.cs.wisc.edu/condor
Campus Scientific Computing
Support the increasing demand from domain scientists for advanced computing, storage and networking services Computing power State-of-the-art middle-ware and libraries Access to experts who understand the nature
and dynamics of scientific computing Cycles for class/research projects
![Page 22: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/22.jpg)
www.cs.wisc.edu/condor
Software Distribution and
Support
Making software developed in academia available to academic and commercial users. Legal and technical support for
software distribution Infrastructure for “for-fee” support Blueprint for dealing with IP rights
![Page 23: Condor : A Concept, A Tool and A Model](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56814737550346895db4763b/html5/thumbnails/23.jpg)
www.cs.wisc.edu/condor
Do not bepicky, be
agile!!!