An Adaptive Semantic Filter for Blue Gene/L Failure Log Analysis

Yinglung Liang Yanyong ZhangECE Dept.

Rutgers University{ylliang,yyzhang} wece.rutgers.edu

Hui XiongMSIS Dept.

Rutgers Universityhui wrbs.rutgers.edu

Ramendra SahooIBM T. J. WatsonResearch Center

rshaoo wus.ibm.com

Abstract- Frequent failure occurrences are becoming a seriousconcern to the community of high-end computing, especiallywhen the applications and the underlying systems rapidly growin size and complexity. In order to better understand the failurebehavior of such systems and further develop effective fault-tolerant strategies, we have collected detailed event logs fromIBM Blue Gene/L, which has as many as 128K processors, and iscurrently the fastest supercomputer in the world. Due to the scaleof such machines and the granularity of the logging mechanisms,the logs can get voluminous and usually contain records whichmay not all be distinct. Consequently, it is crucial to filter theselogs towards isolating the specific failures, which can then be use-ful for subsequent analysis. However, existing filtering methodseither require too much domain expertise, or produce erroneousresults. This paper thus fills this crucial void by designing anddeveloping an Adaptive Semantic Filtering (ASF) method, whichis accurate, light-weight, and more importantly, easy to automate.Specifically, ASF exploits the semantic correlation between twoevents, and dynamically adapts the correlation threshold basedon the temporal gap between the events. We have validated theASF method using the failure logs collected from Blue Gene/Lover a period of 98 days. Our experimental results show thatASF can effectively remove redundant entries in the logs, andthe filtering results can serve as a good base for future failureanalysis studies.

I. INTRODUCTION

Meta-scale scientific and engineering applications have beenplaying a critical role in many aspects of the society, suchas economies of countries, health development, and mili-tary/security. The large processing and storage demands ofthese applications have led to the development and deploy-ment of IBM Blue Gene/L, the fastest supercomputer on theTOP500 supercomputer list [2]. Blue Gene/L is currently de-ployed at Lawrence Livermore National Laboratory (LLNL),hosting applications that span several thousand processors, inthe domains such as hydrodynamics, molecular dynamics, andclimate modeling.As applications and the underlying platforms scale to this

level, failure occurrences have become a norm, rather thanan exception [21], [19], [20], [24], [10], [7], [17], [12], [11],[25], [18]. Failures can be broadly categorized into two classes:software failures and hardware failures. Software failures canbe further categorized into application software failures such asbugs in application development, and system software failuressuch as bugs in the kernel domain and system configurationerrors. Hardware failures are often observed in memory, stor-age and network subsystems, but more recently they are alsofound in combinational units [14], [15]. Both system softwarefailures and hardware failures can have severe impact on thesystem performance and operational costs. For instance, fail-ures can make nodes unavailable, thereby lowering system uti-

1-4244-0910-1/07/$20.00 ©2007 IEEE

lization. Furthermore, failures can cause applications executingon the nodes (probably having run for a long time) to abort,thus wasting the effort already expended. Additionally, failurescan also greatly increase the system management costs. Thesystem administrator may need to detect failures, diagnosethe problem, and figure out the best remedial actions. Onthe hardware end, this may entail resetting a node, changingthe motherboard/disk, etc., and on the software end it mayrequire migrating the application, restarting the application, re-initializing/rejuvenating [23] a software module, etc. Indeed,the resulting personnel involvement will increase the TotalCost of Operation (TCO), which is becoming a serious concernin numerous production environments [8], [1].

Understanding the failure behavior in large scale parallelsystems is crucial towards alleviating the above problems.This requires continual online monitoring and analysis ofevents/failures on these systems over long periods of time.The failure logs obtained from such analysis can be useful inseveral ways. First, the failure logs can be used by hardwareand system software designers during early stages of machinedeployment in order to get feedback about system failuresand performance. It can also help system administrators formaintenance, diagnosis, and enhancing the overall health (up-time). Finally, it can be useful in fine-tuning the runtimesystem for checkpointing (e.g. modulating the frequency ofcheckpointing based on error rates), job scheduling (e.g.allocating nodes which are less failure prone), network stackoptimizations (e.g. employing different protocols and routesbased on error conditions), etc.

While failure logs have the above-mentioned potential uses,the raw logs cannot be directly used. Instead, redundant and/orunimportant information must be first removed. There areseveral reasons for doing so. First, there are many recordedwarnings that do not necessarily lead to a failure. Suchwarnings need to be removed (note that a sequence ofwarningswhich do lead to a failure should remain). Second, the sameerror could get registered multiple times in the log, or couldget flagged in different ways (e.g. network busy and messagedelivery error). For instance, we have noticed that a singleEDRAM error on Blue Gene/L is record in 20 successiveentries, each with a different fatal error code. Consequently,it becomes imperative to filter this evolving data and isolatethe specific events that are needed for subsequent analysis.Without such filtering, the analysis can possibly lead towrong conclusions. For instance, the same software error canmaterialize on all nodes that an application is running on, andcan unduely reduce the Mean Time Between Failures (MTBF)than what is really the case.We have obtained event logs containing all the RAS (reli-

I

2

ability, availability, and serviceability) data from 08/02/05 to11/18/05 from IBM Blue Gene/L, with a total of 1,318,137 en-tries. Filtering raw event logs from large-scale parallel systemssuch as Blue Gene/L is a challenging task, mainly due to thelarge volume of the data, the complexity of the systems, andthe nature of parallel applications. While the spatio-temporalfiltering (STF, in short) method that was proposed in [12] hasbeen shown to have the potential to filter out many redundantfatal events, it has the following obvious problems. First, itrequires extensive domain knowledge in the organization of thehardware platform, as well as the logging procedure. Second,it involves a considerable amount of manual effort from thesedomain experts. Third, it imposes challenges in the area ofsoftware engineering because STF involves several passes, andafter each pass, we need to manually manipulate the partialresults for the next usage. Due to these drawbacks, STF is notsuitable for online filtering, which is important for many futureapplications of failure data. Therefore, there is an urgent needto develop some alternative methods to meet these challenges.

To this end, we propose an Adaptive Semantic Filtering(ASF) method. The key idea behind ASF is to use semanticcontext of event descriptions to determine whether two eventsare redundant. In light of this, ASF involves three steps.Firstly, it extracts all the keywords from event descriptionsand builds a keyword dictionary. Secondly, it converts everyevent description into a binary vector, with each element rep-resenting whether the description contains the correspondingkeyword. Using these vectors, we can compute the correlationbetween any two events. Thirdly, it determines whether twoevents are redundant based on their correlation as well as thetemporal gap between them. Specifically, the choice of thecorrelation threshold is not uniform, but varies according tothe time window between two events. For example, two eventsthat are temporally close are considered redundant even withlow correlation, but two far-away events are only consideredredundant when their correlation is very high. Compared toSTF and other existing filtering tools, ASF considers bothsemantic correlation and temporal information, and therefore,the filtering results more accurately capture the real-world sit-uations. More importantly, ASF does not require much humanintervention, and can thus be easily automated. Consequently,ASF is one big step forward towards self-managing onlinemonitoring/analysis for large-scale systems.

In this study, we have applied ASF on the failure logsfrom IBM Blue Gene/L. Our experimental results show thatASF is more accurate than STF in filtering fatal events dueto its consideration of semantic correlation between events.Also, due to its low overhead, we can use ASF to filter non-fatal events as well. We find that ASF is quite effective forall the severity levels, with the resulting compression ratioalways below 30 o, and often below 1%. After merging filteringresults for all severity levels, we find that events naturallyform "clusters", with each clustering having non-fatal eventsfirst, and then one or more fatal events. These clusters canhelp visualize how events evolve with increasing severity.Indeed, this can serve as a good basis for further analysisand investigations.

Overview: The rest of this paper is organized as follows.Section II summarizes the event data logs we have collectedfrom Blue Gene/L. Both the STF and ASF methods arepresented in Section III, and the filtering results are discussedin Section IV. Following these discussion, Section V includesthe related work in filtering event logs. Finally, section VI con-cludes with a summary of the results and identifies directionsfor future work.

II. AN OVERVIEW OF IBM BLUE GENE/L RAS EVENTLOGS

IBM Blue Gene/L has 128K PowerPC 440 700MHz pro-cessors, which are organized into 64 racks. Each rack consistsof 2 midplanes, and a midplane (with 1024 processors) is thegranularity of job allocation. A midplane contains 16 nodecards (which houses the computing chips), 4 I/O cards (whichhouses the I/0 chips), and 24 midplane switches (throughwhich different midplanes connect). RAS events are loggedthrough the Central Monitoring and Control System (CMCS),and finally stored in a DB2 database. The logging granularityis less than 1 millisecond. More detailed descriptions of theBlue Gene/L hardware and the logging mechanism can befound in our earlier paper [12].We have been collecting RAS event logs from Blue Gene/L

since August 2, 2005. Up to the date of November 18,2005, we have totally 1,318,137 entries. These entries arerecords of all the RAS related events that occur within variouscomponents of the machine. Information about scheduledmaintenances, reboots, and repairs is not included. Each recordof the logs has a number of attributes. The relevant attributesare described as follows.

. RECID is the sequence number of an error entry, whichis incremented upon each new entry being appended tothe logs.

* EVENT TYPE specifies the mechanism through which theevent is recorded, with most of them being through RAS[5].

. SEVERITY can be one of the following levels - INFO,WARNING, SEVERE, ERROR, FATAL, or FAILURE -which also denotes the increasing order of severity.

. FACILITY denotes the component where the event isflagged, which is one of the following: LINKCARD,APP, KERNEL, HARDWARE, DISCOVERY, CMCS,BGLMASTER, SERV NET or MONITOR.

. EVENT TIME is the time stamp associated with thatevent.

. JOBID denotes the job that detects this event. This fieldis only valid for those events reported by computing/10chips.

. LOCATION of an event (i.e., which chip/service-card/node-card/link-card experiences the error), can be spec-ified in two ways. It can either be specified as (i) acombination of job ID, processor, node, and block, or(ii) through a separate location field. We mainly use thelatter approach (location attribute) to determine where anerror takes place.

. ENTRY-DATA gives a description of the event.

3

III. FILTERING METHODS

In this section, we present two event filtering methods.First, we briefly introduce the Spatio-Temporal Filtering (STF)method that was proposed in [12]. Then, we propose anAdaptive Semantic Filtering (ASF) method, which exploits thesemantic correlations between events, and can help automatethe filtering process for large systems such as IBM BlueGene/L.

A. A Spatio-temporal Filtering Method

In our previous work [12], we have developed a Spatio-Temporal Filtering (STF) method for parsing Blue Gene/Levent logs and filtering out redundant/unimportant eventrecords. STF involves three steps: (1) extracting and cat-egorizing failure events; (2) performing temporal filteringto compress events from the same chip locations; and (3)performing spatial filtering to coalesce records of the sameevent across different locations.

First, the raw logs have to be preprocessed, such as re-formatting the entries and handling missing attributes. Afterthe preprocessing step, the next step is to extract all theevents with FATAL severity. As pointed out in Section II,an event can be associated with six levels of severity, andSTF is designed to focus on filtering events with severitylevel FATAL because these events can terminate job exe-cutions and thus have the most severe impact on systemperformance. Once all the FATAL events are extracted, basedon the involved hardware components, they are categorizedinto the following six groups: memory related failures (mem),network related failures (net), midplane switch related failures(mps), application I/0 related failures (aio), node card relatedfailures (nc), and unknown failures. The unknown categoryincludes those FATAL events that do not have self-explanatoryentry-data fields. In order to correctly categorize an event,we have to examine its entry data field carefully, and oftena domain expert is needed. After the categorization step, atemporal filtering is conducted at every chip location, withfailures that are from the same job and are close to eachother coalesced into one record, and the filtering results fromdifferent locations are merged in the temporal order usingthe sort-merge method. The temporal filtering is a simplethreshold-based scheme, and the threshold is chosen withthe help of domain knowledge. Finally, due to the parallelnature of these systems and applications, an event may bereported by multiple locations at the same time. Therefore,we adopt a spatial filtering after the temporal filtering phase,which removes redundant records from different locations.Like the temporal filtering step, spatial filtering also employsa threshold, which is again determined by domain experts.

B. An Adaptive Semantic Filtering Method

While STF can effectively compress Blue Gene/L data logs,as shown in [12], it has the following drawbacks. First, itrequires extensive domain knowledge, both when categorizingfatal events and when adopting suitable threshold values.Second, it requires manual operations. For example, upon the

addition of a new event entry, a human operator is needed tocategorize it into different types. As another example, aftereach step, manual operations are needed to process the partialresults to enable operations in the next step. Third, STF onlyemploys simple thresholding-techniques, which cannot handlemany tricky situations, and may lead to incorrect results.

To address these challenges, in this paper, we propose anAdaptive Semantic Filtering (ASF) method, which exploitsthe semantic context of the event descriptions for the filteringprocess. ASF involves the following three steps: (1) building adictionary containing all the keywords that appear in event de-scriptions, (2) translating every event description into a binaryvector where each element of the vector represents whetherthe corresponding keyword appears in the description or not,and (3) filtering events using adaptive semantic correlationthresholds.

1) Keyword Dictionary: The keyword dictionary is thebase for developing the ASF method. The keywords in thedictionary should capture the semantic context of all theevents. Building the dictionary is an iterative process. In eachiteration, we examine an event entry, identify its keywords, andappend new keywords into the dictionary. In order to identifythe keywords of an event description, we have adopted thefollowing removal/replacement rules:

1) Remove punctuation, equal signs, single quotes, doublequotes, parentheses (including the content in the paren-theses).

2) Remove indefinite articles a, an, and definite articlethe.

3) Remove words such as be, being, been, is, are,was, were, has, have, having, do or done.

4) Remove prepositions such as of, at, in, on, upon,as, such, after, with, from, to, etc.

5) Replace an alphabetic-numeric representation of a rack,a midplane, a link card, or a node card by the keywordLOCATION.

6) Replace an eight-digit hexadecimal address by the key-word 8DigitHex Addr.

7) Replace a three-digit hexadecimal address by the key-word 3DigitHex Addr.

8) Replace an eight-digit hexadecimal value by the key-word 8DiqitHex.

9) Replace a [wo-digit hexadecimal value by the keyword2DigitHex.gelace a numeric number by the keyword NUMBER.

11 Replace binary digits by the keyword BinaryBits.12 Replace a register, e.g. rOO, r23, etc., by the keyword

REGISTER.13) Replace a file directory or an image directory by the

keyword DIRECTORY.14) Replace uppercase letters by the corresponding lower-

case letters.15) Replace present participles and past participles of verbs

by their simple forms, e.g. Failing, Failed beingreplaced by Fail.

16) Replace a plural noun by its single form, e.g. errorsbeing replaced by error.

17) RepLace week days and months by the keyword DATE.

After processing all 1,318,137 entries in the raw logs fromAugust 8, 2005 to November 18, 2005, we have identified667 keywords. One of the advantages of this method is thatupon the arrival of new data, we only need to process the newentries as described above, without affecting the earlier data

4

in any way.2) Correlation Computation: Following the construction of

the keyword dictionary, we next convert each event descriptioninto a binary vector for the purpose of semantic correlationcalculation. Suppose there are N keywords. Then the vectorwill have N elements, with each element corresponding to onekeyword. In this way, assigning vectors to event descriptionsbecomes straightforward: 1 denoting the description includesthe associated keyword, and 0 denoting otherwise. In order tomake this step more automatic, we can choose a reasonablylarge value for N so that adding new logs will not requirere-doing the translations for earlier logs, even when these newlogs may introduce new keywords. This approach is furthersupported by the observation that the number of raw eventsmay be huge, but the number of keywords stays more or lessconstant after it reaches a certain level.

After generating a binary vector for event record, we canthen compute the correlation between any two events usingthe a correlation coefficient [16], the computation form ofPearson's Correlation Coefficient for binary variables.

A0

Column Total

B0 1

P (00) P (01)

P (10) P ( 1)

P (+o) P(+I)

Row Total

P (0+)

P(1+)

N

Fig. 1. A two-way table of item A and item B.

For a 2 x 2 two-way table as shown in Figure 1, thecalculation of the X correlation coefficient reduces to

(00)P(11)- P(01)P(10) (1)

where P(ij) denotes the number of samples that are classifiedin the i-th row and j-th column of the table. Furthermore,we let P(i+) denote the total number of samples classified inthe ith row, and P(+j) the total number of samples classifiedin the jth column. Thus, we have P(i+) = E1=o P(ij) and

P(+j) = El oP(ij). In the two-way table, N is the totalnumber of samples.

3) Adaptive Semantic Filtering: ASF tells whether a recordis redundant or not based on its semantic context. An intu-itive semantic correlation based approach would regard tworecords with a high correlation between their descriptions as

redundant. A closer look at the logs, however, reveals that inaddition to the correlation coefficient, the interval between tworecords also plays an important role in determining whetherthese two records are redundant. For example, if two eventsare very close to each other, even though their correlationmay be low, they may still be triggered by the same failure.On the other hand, two records that are far away from eachother, though their descriptions may be exactly the same,are more likely to report unique failures. As a result, it isinsufficient to adopt a simple thresholding technique solelybased on the correlation coefficients between events. Instead,we propose to adopt an adaptive semantic filtering mechanism,which takes into consideration both semantic correlation andtemporal information between events to locate unique events.

26X10'

2.4

2.2-

g2-

° 1.8-

E1.6

1 .4-

1.2 90 0 -0 500 1000 1500 2000 2500 3000 3500 4000 05001000

S....d S....d

(a) The number of fatal (b) The numbeievents after temporal filter- events after spatiing with different tth val- with different st,ues

Fig. 2. Choosing appropriate values of tth and Sth for STF.

1500 2000

r of fatal-al filteringh values

We thus believe that, by doing so, we can combine theadvantage of STF and the advantage of a simple semanticfiltering technique. The principle of the adaptive semanticfiltering is that as the time gap between records increases,two records must have a higher correlation coefficient to beconsidered redundant, and after the time gap reaches a certainlevel, even two records with a perfect correlation coefficientwill be considered unique.

The Adaptive Semantic Filtering (ASF) algorithm first sortsthe log entries in time sequence and stores them into the joblDsymbolic reference table as shown in Line 2 and Line 3. Then,for each anchor event in the joblD table, if the timestampof the current anchor event is greater than the largest timethreshold, the ASF algorithm deletes this anchor event fromthe joblD table; otherwise, in line 8, the algorithm computesthe semantic correlation between the current event and theanchor event. Line 9 obtains the temporal gap between thecurrent event and the anchor event. If either the temporal orthe correlation criteria is satisfied, the current record is filteredout as indicated in Line 12. The above process is iterated forevery event record.

IV. EXPERIMENTAL RESULTS

In this study, we have applied the proposed adaptive se-mantic filtering technique to the raw RAS event logs collectedfrom the Blue Gene/L machine. We report the detailed filteringresults in this section.

A. ASF Versus STF

First, we would like to compare the effectiveness of thenew semantic filter ASF, and the earlier non-semantic filterSTF. Please note that STF is expected to accurately extractunique events from voluminous raw logs because it hasinvolved extensive domain knowledge in the organization ofthe hardware platform, as well as the logging procedure, andbecause it has involved a considerable amount of manual effortfrom these domain experts.Due to the space limit, instead of presenting the filtering

results for all types of events, we only present the results forFATAL events here. For the sake of fairness, we first need tocarefully tune the parameters of both algorithms to reach theiroptimal operating ranges. The main parameters involved in

1800

1700

1600tI::. 1500I

1400

2 1300

1200Ez

1100

i000

5

STF include the threshold in the temporal filtering phase tth,and the threshold in the spatial filtering phase Sth. Figure 2(a)plots the number of remaining FATAL events after applyingdifferent temporal values tth. We take the viewpoint that ajob is likely to encounter only one fatal event of the sametype. Hence, a threshold that is too small will result in severalfatal events from the same type for a job. On the other hand,since the log has a large portion of entries that do not havea valid JobID field, then a large threshold may filter awayfailures encountered by different jobs. Both factors considered,we have chosen tth = 20 minutes, and this threshold yields145371 fatal events. Of course, after choosing this threshold,we have validated our choice by examining both the originallog and the resulting log manually.Compared to the temporal threshold, the spatial threshold

is easier to set. Similarly, Figure 2(b) gives the number ofremaining fatal events after applying different spatial filteringthreshold values Sth. We have two main observations from thisfigure. First, applying spatial filtering is very important. Evena zero-second spatial filtering threshold can bring down thenumber of FATAL events from 145371 to 1746. The secondobservation is that, the impact of different spatial thresholdvalues is not as pronounced as that of the temporal filteringthreshold. This is because the fact that spatial filtering isadopted dominates the filtering effect. As a result, we choose20-minute as the value for Sth.

Using the chosen threshold values, STF can bring down thenumber of FATAL records from 281462 to 998, which onlyconstitutes 0.3546% of the raw log.Now, let us switch our attention to the proposed adaptive

semantic filter (ASF). As presented in Section III, ASF adoptsdifferent correlation coefficient threshold values according tothe intervals between subsequent event records. Specifically,we take the viewpoint that two records that are temporallyclose to each other are likely to be correlated, and therefore,should be coalesced into one event. As a result, we adopta lower correlation threshold for shorter intervals betweensubsequent records. On the other hand, two records that arefar apart from each other should only be considered correlatedwhen the semantic correlation between them is high, whichsuggests that we should adopt a higher threshold for eventswith larger intervals from their preceding events.

In order to develop suitable threshold values, we havepartitioned the data sets into two halves, the first half beingtraining data while the second half being test data. On thetraining data, we have applied different correlation coefficientand interval pairs, and chosen the following values which haveproduced similar results as those from STF:

Cth =

10.0.0.0.

if T C [20, co)9 else ifT c [10, 20)8 else if T C [5, 10)7 elseifT[1,5)0 else if T C [0.5, 1)1.0 else if T e [0, 0.5)

(2)

where T denotes the interval between the current record andthe previous record, and the time unit is a minute. Equation 2specifies correlation coefficient threshold values for different

intervals. For example, if the gap between the current recordand the previous record, T, is greater than or equal to 20minutes, then the current record will be kept in the resultlog if the semantic correlation between it and its previousrecord is less than or equal to 1.0. (Of course, since 1.0 isthe maximum correlation coefficient, all the events that occurwithin a window longer than 20 minutes after their precedingevents will be kept.) As another example, Equation 2 specifiesthat if T is less than 30 seconds, then the current event willbe filtered out if the semantic correlation between itself andits previous event is less than -1.0. In another word, all theevents that occur within 30 seconds after their previous eventswill be filtered out.

After we extract the parameters in Equation 2 from thetraining data, we have applied them to the test data to examinewhether they are only specific to the training data or they canbe used to the test data as well. Fortunately, we find that thesevalues are effective for all the data after careful inspection.

Using the chosen parameters, ASF can condense the fatalfailures from 281462 records to 835 records, while STF pro-duces 998 records after filtering. Though these two numbersare rather close, we have observed three typical scenariosin which these two filters yield different results, and thatamong these three, two cases demonstrate ASF produces betterfiltering results than STF. These four scenarios are listed below(each record contains the timestamp, job ID, location, andentry data fields):

Advantage I. ASF can efficiently filter out semantically-correlated records whose descriptions do not exactlymatch each other Records that are semanticallycorrelated should be filtered out (if they are reasonablyclose to each other), even though they do not haveidentical descriptions. ASF can easily achieve thisbecause it considers semantic correlation betweenevents. STF, on the other hand, compares eventdescriptions word by word, which can be problematicbecause many highly-correlated records do not haveidentical descriptions. For example, STF has producedthe following records in its result:

[ST1]2005-11-15-12.07.42.786006 - R54-M0-N4-I:J1 8-Ul1LOGIN chdir(/xxx/xxx) failed: No such file or directory[ST2]2005-11-15-12.07.42.858706 - R64-M1-N8-I:J18-Ul1LOGIN chdir(/xxx/xxxx/xx) failed: No such file or directory

[ST3]2005-11-15-12.07.42.779642 - R74-M0-NC-I:J18-Ul1

LOGIN chdir(/xxx/xxxx/xxx) failed: No such file or directory

ciod:

ciod:

ciod:

In this example, all three events occur at the same time,but at different locations, and they correspond to the samefatal failure that affects all three locations. However, inthe spatial filtering phase, STF only filters out records iftheir descriptions are the same. As a result, it has kept allthree entries. This problem, however, can be avoided byASF because ASF considers semantic correlation insteadof exact word-by-word match. Hence, the result fromASF only contains one entry:

[AS 1 ]2005- 11-15-12.07.42.786006 - R54-MO-N4-I: J1 8-Ul1 ciod:

LOGIN chdir(/xxx/xxx) failed: No such file or directory

6

This example emphasizes the importance of semanticcorrelations in filtering error logs.Advantage II. ASF can prevent non-correlated eventsfrom beingfiltered out. The previous example shows thatASF can filter out semantically correlated events evenwhen their descriptions are not identical. Similarly, ASFcan also prevent non-correlated events from being blindlyfiltered out just because they are close to each other. Thisis because STF, in its temporal filtering phase, simplytreats all the events that are more than 20 minutes apart asunique while all the events that are less than 20 minutesapart as redundant. Compared to STF, ASF employs amuch more sophisticated mechanism, which not onlyexploits correlation coefficient between two events, andthe threshold for the correlation coefficient also adaptsto the gap between the events. As a result, if the gapbetween two events (from the same location) is less than20 minutes, STF will filter out the second event, but ASFwill only do so if their correlation coefficient is above aceratin level.As an example, ASF has produced the following se-quence:[AS1]2005-09-06-08.45.57.171235 - R62-MO-NC-I:J1 8-Ul1 ciod:LOGIN chdir(/home/xxx/xx/run) failed: Permission denied

[AS2] 2005-09-06-08.49.34.442856 - R62-MO-NC-I:J18-U11

ciod: Error loading "Ihomelxxxxxlxxxlxxxxlxxxx": program image too big,

1663615088 > 532152320

In the above sequence, ASF chooses to keep both recordsbecause the semantic correlation between them is lessthan 0.0, and according to parameters in Equation 2, theyare unique events. On the other hand, STF condenses thesame example scenario to the only entry because the gapbetween these two events is 4 minutes:

[ST 1 ]2005-09-06-08.45.57. 17123 5 - R62-MO-NC-I:J1 8-Ul1 ciod:

LOGIN chdir(/home/xxx/xx/run) failed: Permission denied

Comparing the results produced by both filters, we caneasily tell that both entries need to be kept because eachcorresponds to a different problem in the system.Disadvantage: ASF may filter out unique events thatoccur with 30 seconds from each other According toEquation 2, ASF filters out events when they occur within30 seconds of each other. Though in most cases, eventsthat are so close to each other are highly correlated, thereare some rare cases where different types events maytake place at the same time, e.g. a memory failure and anetwork failure occurring at the same time. STF can avoidsuch problems by categorizing failures before filtering.For example, STF has produced the following sequenceof records:[ST1]2005-08-05-09.11.35.447278 - R33-M1-NC-C:J13-U11 ma-chine check interrupt

[ST2]2005-08-05-09.11.59.393092 - R54-M1-N8-I:J18-UO1 ciod:

Error reading message prefix after LOADMESSAGE on CioStream socket to

xxx.xx.xx.xxx:xxxxx: Link has been severed

Info Warning Severebefore filtering 1,367,531 17,121 15,749after filtering 11,044 343 148

compression ratio 0.008 0.02 0.009Error Failure Fatal

before filtering 109,048 1,708 281,441after filtering 146 53 1,147

compression ratio 0.001 0.03 0.004

TABLE I

THE NUMBER OF EVENTS AT ALL SEVERITY LEVELS BEFORE AND AFTER

FILTERING BY ASF.

Since these two failures, one being memory failure andthe other application I/0 failure, occur within 24 secondsfrom each other, ASF compresses them into one entry:

[ST1]2005-08-05-09.11.35.447278 R33-M1-NC-C:J13-U11 ma-

chine check interrupt

Fortunately, this problem of ASF does not affect thefiltering results much because the likelihood of havingtwo failures within 30 seconds is very low. In fact, wehave checked the log carefully, and found that the aboveexample is the only case where two distinct events occurso close to each other. Even in such cases, the problemwill be further alleviated by the fact that the productionBlue Gene/L usually runs one job at a time, which spansall the processors of that machine. As a result, the adverseeffect of compressing failures that occur at the same timeis negligible because they hit the same job anyway.

B. The Blue Gene/L RAS Event Analysis

In addition to filtering fatal events, it is also equally impor-tant to filter other non-fatal events, since such information candepict a global picture about how warnings evolve into fatalfailures, or about how a fatal failure is captured by differentlevels of logging mechanisms. STF, however, cannot be usedto filter non-fatal events due to its complexity, especiallywhen the number of non-fatal events (1,172,766 in our case)is substantially larger than that of fatal events (281,462).Fortunately, this void can be filled by the introduction of ASF,which involves much less overhead and can thus yield anautomatic execution.

Table I summarizes the filtering results for events at allseverity levels. These numbers show that ASF is quite effectivein filtering all types of events, achieving compression ratiosbelow 300 (many compression ratios are below 1%). Afterfiltering events of all severity levels, we can next merge themin the temporal order, and study how lower-severity eventsevolve to a FAILURE or FATAL event, which can terminatejob executions and cause machine reboots. We would notethat, the investigation of detailed rules about what non-fatalevents will lead to fatal failures, and in what fashion, is wellbeyond the scope of this paper. In this paper, we argue thatsuch studies are made possible by the introduction of ASF.

After merging events with different severity levels, weobserve that they form natural "clusters" consisting of mutiplenon-fatal events and one or more fatal events following them.These clusters clearly show that how events evolve in their

facility severity timestampCMCS INFO 2005-11-07-08.40.12.867033HARDWARE WARNING 2005-11-07-08.40.48.975133

DISCOVERY WARNING 2005- 11-07-08.42.07.610916DISCOVERY SEVERE 2005-11-07-08.42.07.769056DISCOVERY ERROR 2005-11-07-08.42.07.797900

HARDWARE SEVERE 2005-11-07-12.28.05.800333MONITOR FAILURE 2005-11-07-14.11.44.893548HARDWARE SEVERE 2005-11-07-14.38.38.623219

location entry data- Starting SystemController UNKNOWNlOCATIONR63-MO EndServiceAction is restarting the NodeCards in midplane R63-MO as part

of Service Action 541R63-MO-N6 Node card is not fully functionalR63-MO-N6 Can not get assembly infonnation for node cardR63-MO-N6 Node card status: no ALERTs are active. Clock Mode is Low. Clock

Select is Midplane. Phy JTAG Reset is asserted. ASIC JTAG Reset is

asserted. Temperature Mask is not active. No temperature error. TemperatureLimit Error Latch is clear. PGOOD IS NOT ASSERTED. PGOOD ERRORLATCH IS ACTIVE. MPGOOD IS NOT OK. MPGOOD ERROR LATCHIS ACTIVE. The 2.5 volt rail is OK. The 1.5 volt rail is OK.

R63-MO-L2 LinkCard power module U58 is not accessibleR63-MO-L2 No power module U58 found found on link cardR63-MO-L2 LinkCard power module U58 is not accessible

TABLE II

AN EXAMPLE EVENT SEQUENCE THAT REVEALS HOW INFO EVENTS EVOLVE INTO FAILURE EVENTS.

severity. An example cluster is shown in Table II. Thissequence starts with an INFO event that informs the systemcontroller was starting. Thirty seconds later, all the node cardson midplane R63-MO were restarted, as suggested in thefollowing WARNING event, and another two minutes later,another WARNING message points out that one of the nodecards on that midplane, R63-MO-N6, was not fully functional.At almost the same time, a SEVERE event and an ERRORevent were recorded, which give more detailed informationabout the node card malfunction. The SEVERE event reportsthat the assembly information could not be obtained for thesame node card, and the ERROR event reports several majorstatus parameters of the node card, such as "PGOOD is notasserted", "MPGOOD is not OK", etc. About 4 hours later,a SEVERE event reports that one of the link cards' power

module U58 was not accessible from the same midplane, andabout 2 hours later, the power module U58 was reported totallyun-found by a FAILURE event. After the FAILURE event, themidplane needs to be repaired by a system administrator beforeit can be used again.

The ability to locate such sequences is important forstudying failure behavior and predicting failures. This was

impossible without a good filtering tool. In our example above,there are only 8 records, but they correspond to a much longersequence in the raw logs, with 572 records. We would notethat, it is very difficult, if not at all impossible, to keep trackof event occurrences from a 572-entry sequence.

V. RELATED WORK

Collection and filtering of failure logs has been examinedin the context of small-scale systems. For example, Lin andSiewiorek [13] found that error logs usually consist of morethan one failure process, making it imperative to collect theselogs over extended periods of time. In [3], Buckley et al.made recommendations about how to monitor events andcreate logs by using one of the largest data sets at that time,comprising 2.35 million events from a VAX/VMS systemwith 193 processors. They pointed out that data sets withpoor quality are not very helpful, and can lead to wrong

conclusions. Their findings reiterated several important issuesin this area, namely, lack of information in the logs (e.g.power outages), errors in the monitoring system (e.g. in thetimestamps), and the difficulty of parsing and collecting useful

patterns. It has also been recognized [4], [6], [9], [22] thatit is critical to coalesce related events since faults propagatein the time and error detection domain. The tupling conceptdeveloped by Tsao [22] groups closely related events, andis a method of organizing the information in the log into a

hierarchical structure to possibly compress failure logs [4].Filtering raw logs becomes more important for larger par-

allel and distributed systems. In our previous study [17], we

studied the failure behavior for a large-scale heterogeneousAIX cluster involving 400 nodes over a 1.5 year period. Inthat study, we used a simple thresholding technique to filter outredundant entries, and the threshold was 5 minutes. In anotherprevious study [12], we developed a spatio-temporal tool(STF) to filter logs collected from a Blue Gene/L prototypeconsisting of 8192 processors. STF was the first filtering toolthat could deal with large failure data sets, and is used as thebaseline technique in this exercise.

VI. CONCLUSIONS AND FUTURE WORK

Parallel system event/failure logging in production envi-ronments has widespread applicability. It can be used toobtain valuable information from the field on hardware andsoftware failures, which can help designers make hardware andsoftware revisions. It can be used by system administrators fordiagnosing problems in the machine, scheduling maintenanceand down-times. Finally, it can be used to enhance faultresilience and tolerance abilities of the runtime system fortuning checkpointing frequencies and locations, parallel jobscheduling, etc. With fine-grain event logging, the volume ofdata that is accumulated can become unwieldy over extendedperiods of time (months/years), and across thousands of nodes.Further, the idiosyncracies of logging mechanisms can lead tomultiple records of the same events, and these need to becleaned up in order to be accurate for subsequent analysis.

In this paper, we have presented an Adaptive SemanticFiltering (ASF) method, which exploits the semantic corre-

lation as well as the temporal information between eventsto determine whether they are redundant. The ASF methodinvolves three steps: first building a keyword dictionary, thencomputing the correlation between events, and finally choosingappropriate correlation thresholds based on the temporal gap

between events. Compared to existing filtering tools, theproposed filter (1) produces more accurate results, (2) incurs

7

8

less overhead, and (3) avoids frequent human intervention.We have validated the design of the filter using the failurelogs collected from Blue Gene/L, which consists of 128Kprocessors, and is the fastest supercomputer on the Top 500Supercomputers List, over a period of 98 days.

Fault-tolerance for large-scale systems requires long-termefforts from the entire community, and this study only servesas a starting point towards this goal. There are several in-teresting possibilities for future work, and we are particularlyinterested in investigating online statistical analysis of this datafor predictability. Also, we are planning to use this informationfor enhancing the runtime fault-tolerance mechanisms such ascheckpointing and job scheduling.

REFERENCES

[1] The UC Berkeley/Stanford Recovery-Oriented Computing (ROC)Project. http: //roc.cs.berkeley.edu/.

[2] TOP500 List 11/2004. http://www.top500.org/lists/2005/11/basic.[3] M. F. Buckley and D. P. Siewiorek. Vax/vms event monitoring and

analysis. In FTCS-25, Computing Digest of Papers, pages 414-423,June 1995.

[4] M. F. Buckley and D. P. Siewiorek. Comparative analysis of eventtupling schemes. In FTCS-26, Computing Digest ofPapers, pages 294-303, June 1996.

[5] M. L. Fair, C. R. Conklin, S. B. Swaney, P. J. Meaney, W. J. Clarke, L. C.Alves, I. N. Modi, F. Freier, W. Fischer, and N. E. Weber. Reliability,Availability, and Serviceability (RAS) of the IBM eServer z990. IBMJournal of Research and Development, 48(314), 2004.

[6] J. Hansen. Trend Analysis and Modeling of Uni/Multi-ProcessorEvent Logs. PhD thesis, Dept. of Computer Science, Carnegie-MellonUniversity, 1988.

[7] T. Heath, R. P. Martin, and T. D. Nguyen. Improving cluster availabilityusing workstation validation. In Proceedings of the ACMSIGMETRICS2002 Conference on Measurement and Modeling of Computer Systems,pages 217-227, 2002.

[8] IBM. Autonomic computing initiative, 2002.http://www.research.ibm.com/autonomic/index nf.html.

[9] R. Iyer, L. T. Young, and V. Sridhar. Recognition of error symptons inlarge systems. In Proceedings of the Fall Joint Computer Conference,1986.

[10] M. Kalyanakrishnam and Z. Kalbarczyk. Failure data analysis of alan of windows nt based computers. In Proceedings of the 18th IEEESymposium on Reliable Distributed Systems, 1999.

[11] Y Liang, Y. Zhang, M. Jette, A. Sivasubramaniam, and R. Sahoo.Bluegene/1 failure analysis and prediction models. In Proceedings of theInternational Conference on Dependable Systems and Networks (DSN),2006. To Appear.

[12] Y Liang, Y. Zhang, A. Sivasubramaniam, R. Sahoo, J. Moreira, andM. Gupta. Filtering failure logs for a bluegene/1 prototype. InProceedings of the International Conference on Dependable Systemsand Networks (DSN), 2005.

[13] T. Y. Lin and D. P. Siewiorek. Error log analysis: Statistical modellingand heuristic trend analysis. IEEE Trans. on Reliability, 39(4):419-432,October 1990.

[14] S.S. Mukherjee, C. Weaver, J. Emer, S.K. Reinhardt, and T. Austin.A Systematic Methodology to Compute the Architectural VulnerabilityFactors for a High-Performance Microprocessor. In Proceedings of theInternational Symposium on Microarchitecture (MICRO), pages 29-40,2003.

[15] A. Parashar, S. Gurumurthi, and A. Sivasubramaniam. A Complexity-Effective Approach to ALU Bandwidth Enhancement for Instruction-Level Temporal Redundancy. In Proceedings of the InternationalSymposium on Computer Architecture (ISCA), 2004.

[16] H. T. Reynolds. The Analysis of Cross-classifications. The Free Press,New York, 1977.

[17] R. Sahoo, A. Sivasubramaniam, M. Squillante, and Y. Zhang. FailureData Analysis of a Large-Scale Heterogeneous Server Environment.In Proceedings of the 2004 International Conference on DependableSystems and Networks, pages 389-398, 2004.

[18] R. K. Sahoo, A. J. Oliner, I. Rish, M. Gupta, J. E. Moreira, S. Ma, R. Vi-lalta, and A. Sivasubramaniam. Critical Event Prediction for ProactiveManagement in Large-scale Computer Clusters In Proceedings of theNinth ACMSIGKDD International Conference on Knowledge Discoveryand Data Mining, August 2003.

[19] D. Tang and R. K. Iyer. Impact of correlated failures on dependability ina VAXcluster system. In Proceedings of the IFIP Working Conferenceon Dependable Computing for Critical Applications, 1991.

[20] D. Tang and R. K. Iyer. Analysis and modeling of correlated failures inmulticomputer systems. IEEE Transactions on Computers, 41(5):567-577, 1992.

[21] D. Tang, R. K. Iyer, and S. S. Subramani. Failure analysis and modellingof a VAXcluster system. In Proceedings International Symposium onFault-tolerant Computing, pages 244-251, 1990.

[22] M. M. Tsao. Trend Analysis and Fault Prediction. PhD thesis, Dept. ofComputer Science, Carnegie-Mellon University, 1983.

[23] K. Vaidyanathan, R. E. Harper, S. W. Hunter, and K. S. Trivedi. Analysisand Implementation of Software Rejuvenation in Cluster Systems. InProceedings of the ACM SIGMETRICS 2001 Conference on Measure-ment and Modeling of Computer Systems, pages 62-71, June 2001.

[24] J. Xu, Z. Kallbarczyk, and R. K. Iyer. Networked Windows NT SystemField Failure Data Analysis. Technical Report CRHC 9808 Universityof Illinois at Urbana- Champaign, 1999.

[25] Y Zhang, M. Squillante, A. Sivasubramaniam, and R. Sahoo. Per-formance Implications of Failures in Large-Scale Cluster scheduling.In Proceedings of the 10th Workshop on Job Scheduling Strategies forParallel Processing, June 2004.