hawkeye [edocfind.com] (1)

8/7/2019 hawkeye [EDocFind.com] (1)

1/25

www.cs.wisc.edu/condor 1

HawkEyeA Monitoring and Management

Tool for Distributed SystemsTodd Tannenbaum

Department of Computer Sciences

University of Wisconsin-Madison http://www.cs.wisc.edu/condor [email protected]


2/25


What does Condor have? lots of c ore technology for bui lding a distributed system


3/25


What does Condor have? lots of core technology for building a distributed system lots of core technology for monitoring

the status of a machine


4/25




lots of core technology for managing a w ork load of tasks


5/25




lots of core technology for managing a w ork load of tasks lots of really, truly, skilled and

experienced developers and researchers at building distributed systems . S ome of the best . S tandout state employees . Honest .E mail for W isconsin G ov S cott McC allum:

w isgov@ gov.state .w i.us


6/25


One day an

avid Condoruser asked:


7/25


One day an

avid Condoruser asked:

Say, could CondorTechnology be used

for distributed system administration??


8/25


Time to think Gathered up our experiences w ith our own management tasks, looked at the mature C ondor technology available to us, and HawkEye effort was born .

Completely separate from Condor from end user prospective .

Can install HawkEye, or Condor, or both


9/25


First Component:MONITORING

Sysadmins first need information about what is happening on the

machines they are responsible for .Both Current and PastInformation must be consolidated and easily accessibleInformation must be dynamic


10/25


Condor ClassAds T echnology for an entity to describe

itself Simple attribute value pairs

[ load_average = 1.3free_Swap_space_mb = 140number_of_processes = 92keyboard_idle_secs = 6ram = 128total_swap = 512total_memory = ram + total_swapbusy = load_average > 1.0

]


11/25


Condor ClassAds, cont. No fixed schema Attributes can contain values or

expressions

Serialize Ads in XML Open source libraries on C++ and J ava to:Manipulate Ads and Ad attributesStore Ads

Query collections of Ads Bindings for Perl and others on the way


12/25


13/25


HawkEye Monitoring Agent


HawkEyeManager HawkEye Monitoring Agent




14/25



/proc, kstat

Hawkeye_Startup_Agent

Hawkeye_Monitor


HawkEyeManager ClassAd

UpdatesViaSecureUDP


15/25


Monitor Agent, cont.

Updates are sent periodicallyInformation does not get stale

Updates also serve as a heartbeat monitorKnow when a machine is down Out of the box, the update ClassAd has

many attributes about the machine of interest for system administration

Current Prototype = 184 attributes


16/25


What if I wantto monitor

something youdidnt think

about?


17/25


Custom Attributes

/proc, kstat

Hawkeye_Startup_Agent

Hawkeye_Monitor


HawkEyeManager

Data fromhawkeye_update_attribute

command line tool

Create your own

HawkEye plugins,or share plugins withothers


18/25


Role of HawkEye

Manager Store all incoming ClassAds in a indexed resident data structure

Fast response to client tool queries about current stateShow me all machines with a load average > 10

Periodically store ClassAd attributes into a Round Robin D atabase

Store information over timeShow me a graph with the load average for this machine over the past week

Speak to clients via CEDAR, HTTP

HawkEyeManager


19/25

S everal different clients Command- line, GUI, W eb - based


20/25


But sysadmins alsosometimes have to do

work T ask: copy a new library onto the

local disk of each machine.J ust a script to copy via rcp/scp to every machine or is it?


21/25


Running tasks on behalf of the sysadmin Submit your sysadmin tasks to HawkEye

T asks are stored in a persistent queue by the Manager

T asks can leave the queue upon completion, or repeat after specified intervalsT asks can have complex interdependencies via DAGManRecords are kept on which task ran where

Sounds like Condor, eh?Yes, but simpler


22/25


Run Tasks in response tomonitoring information ClassAd Requirements Attribute

Example: Send email if a machine is low on disk space or low on swap space

Submit an email task with an attribute: Requirements = free_disk < 5 || free_swap < 5

Example w/ task interdependency: If load average is high and OS=Linux and console is Idle, submit a task which runs top , if top sees Netscape, submit a task to kill Netscape


23/25


24/25


Current S tatus

J ust Beginning this project Initial release early summer Prototypes already running

Stop in and see initial HawkEye W orkRm 3385 on W eds 9am 12pm


25/25

www.cs.wisc.edu/condor25

Thank you!

I was an

overworkedsysadmin. NowI have more free

time thanks toHawkEye!

hawkeye [edocfind.com] (1)

Documents