vast 2008 challenge - university of maryland€¦ · 26/10/2008 · on day 7, this group dropped...

VAST 2008 ChallengeThe VAST 2008 Challenge Co-chairs

Georges Grinstein, University of Massachusetts LowellCatherine Plaisant, University of MarylandJean Scholtz, Pacific Northwest National Laboratory

The VAST 2008 Challenge CommitteeSharon Laskowski, National Institute of Standards and TechnologyTheresa O’Connell, National Institute of Standards and TechnologyMark Whiting, Pacific Northwest National Laboratory

The Challenge Committee wishes to thank

The VAST and VisWeek organizers All participantsAll students who supported the project

Loura Costello, Heather Byrne, Adem Albayrak

The analysts and judgesOur sponsors

Thanks to our Sponsors

NVAC

NSF

NIST

Battelle Memorial Institute

IARPA A-SpaceX

VAST 2008 ChallengeNew format

4 Mini-ChallengesEach has own dataset and tasksTeams may enter one or more

Grand ChallengeAnalyze all 4 datasets – Integrate to answer

No “1st place” – but instead many awards for outstanding aspects of the entriesBoth visualization and analysis awards

The VAST ChallengesDemonstrate the visual analytics capabilities

of your tools against an invented scenario and synthetic datasets.

Challenge themes included geospatial, activity and behavior, text processing, and social network analyses.

Strong InterestOver 400 downloads of 2008 data set

Compared to 150 to date for 2007 data set73 submissions

Cell phone (social network) 22Boats landings (geo spatial) 13Evacuation Traces (geo spatial) 20Wiki pages (finding factions) 12Grand Challenge 6

28 organizations12 student teams13 countries

This morningNow

Overview of datasets, questions, judgingAwards and summary of submissions to each of the 4 mini challenges

After the breakChallenge Panel

Data and interactive sessionPresentation by 3 Grand Challenge teamsQuestions to teamsOpen Discussion

Challenge Scenario: the dataInformation

A web page about the Paraiso movement covering the history and beliefs of the group

Data1. wiki edits2. cell phone call logs3. Coast Guard interdiction records4. records of an evacuation of a Department of

Health Building

Challenge Scenario: the tasksParticipants in each Mini Challenge were required to analyze a single data setsParticipants could enter more than oneParticipants in the Grand Challenge were required to analyze all four data sets

Grand Challenge tasksAssess the beliefs of the movement and the activities they’re involved inDetermine if the movement advocates violence

JudgingJudging was done by a committee of 15

7 members were professional analysts8 members have usability and visualization expertise

Criteria usedUtility in arriving at the answerAccuracy of the answerQuality of the visualizationsFunctionality of the analytic environment

AwardsVisualization awards

System awards

Analysis awards

Award policy – only one award per teamAwards were specific to the various mini challenges and to the quality of the submissions this year. We will not necessarily be giving the same awards next year.

Intuitive visualizationsInnovative visualizationsOutstanding functionalityLevel of integrationAccuracyHigh quality report

Cell Phone Challenge

Cell Phone LogsDataset

10 days of cell phone call logsCaller id / callee id / cell tower(=location) / date / length of call

Analytic SituationA social faction (supporters of the “Paraiso”movement) was a close knit group of individuals who maintained cell call contact. On day 7, this group dropped their phones and picked up another set.

Cell Phone QuestionsWhat is the Catalano/Vidro social network, as reflected in the cell phone call data, at the end of the time period? Characterize the changes in the Catalano/Vidro social structure over the ten day period?

GeoTemporalNet

MobiVis

Oculus Info Inc.

Almost all teams visualized the social network with a node/link diagram

INRIA Bordeaux - VizWhiz

Other Representations

Calls from 400 number to 398 number (123), R2=10%, XY projection

Nyenrode Business Univesiteit University College Dublin -SocialDynamicsVis

Other Representations

CORE

Palantir

Prajna Project

Visualization/Analysis of the geospatial components of the data

ICAVEAttributeLayoutViz


SocialDynamicsVis

PALANTIR

SPADAC

Handling TimeUsing Histograms

University of Maryland

Handling TimeUsing Stacked Filled Curves

VizWhiz

NEVAC

ComVis

SocialDynamicsVis

Coordinated Displays

Oculus Info Inc.


University of IllinoisNational Center for Supercomputing


Cell Phone Awards

Beijing University of Posts and TelecommunicationsSocial Network Accuracy

The submitted social network was analyzed using a computed average of node/link precision and recall. Highest accuracy was 55.9%

University of California, DavisIntuitive Social Network Graphs

University College DublinNode-Link Animation

University of Maryland – SocialActionTime Visualizations of Cell Phone Activity

Vision Systems & Technology, Inc.Effective Toolkit Integration

Boats Challenge

Boat LogsDataset

Logs of landings or interception by Coast GuardLaunch point / landing point / type of vessel / date / passengers / whether or not intercepted

Analytic SituationBoat migrants (Paraiso supporters) were leaving the Isla del Sueno for Florida in massive numbers. The US Coast Guard were picking them up with increasing success. Their landing strategy moved westward until successful landing began on the Yucatan Peninsula, Mexico.

Boat QuestionsCharacterize the choice of landing sites and their evolution over the three year

Characterize the geographical patterns of interdiction over the three years

What is the successful landing rate over the time period

Map or no map?

Grid with lat/long coordinates Beijing

Many used Google EarthSome: own GISSome: no map, or optional.

CMU team

U of Singapore

U of Illinois

Grouping and clustering

SPADAC

ParvacU of Washington

Aggregation (and dealing with overlap)

Parvac – U of WashingtonAggregation by regionX axis is region, e.g. L5 = Mexico

OculusIrregular areaAggregation on the map

Representing TimeAnimation

Those who relied solely on animation missed the large number of Mexico landings (they overlapped in a very small area)

Timelines

VSTI Prajna

3D

Hybrid Projection

Representing Time

Oculus

Tacc

Symbol used on the map

Dots - CORE

Lines - Tacc

Centroid - Tacc

SPADAC - Arrows

Coordinated displays

GET COMVIS EXAMPLE

Zagreb/VrVis/VTech

Palantir

Analysis SupportMany merely reported numbersSome made hypothesesA few provided some analysis or reporting support

Oculus

Analysis thoroughnessSome teams provided the minimum answersOther looked at the data very thoroughlye.g. looking at every angles of the data, exploring seasonal patterns, finding suspects’ names amongst passengers

Recommendation

TaccX is time, Y latitudeBlack lines are coast guard patrol traces

A few gave suggestionse.g. new areas for the Coast Guard to patrol

Boat Awards

SPADAC Inc.Analysis Summary

Summary pointed out that Mexico landings could be seen as a land bridge to Florida and noted that these were successful because US Coast Guard would have no jurisdiction in Mexico.

VRVis – U. of Zagreb – Virginia TechSimple and Effective Integrated Display

Traces Challenge

Traces LogsDataset

RFID logs during a building evacuationX, Y coordinates over time period per ID

Analytic SituationA small improvised explosive device (IED) was set off in a at a (fictitious) Miami, Florida Department of Health (DOH) Building, resulting in casualties and moderate damage. RFID was used to track all employees & visitors. A Paraiso support set off the device, but there were witnesses.

Trace Challenge QuestionsWhere was the device set off? Identify potential suspects or witnessesIdentify any suspects and/or witnesses who managed to escape the building. Identify any casualtiesDescribe the evacuation

Sorting it all out Staining by Southern Illinois

Who did it? Where did they do it? Wisdom of the crowds (users) finger Ramon Katalanow

UBari – IVU Lab

Who saw it?

Anton Knapp (trace visible) turns towards Ramon Katalanow (circled in red) during the evacuation.

Katalanow then moves away from Knapp to other exit.

VSTI-Prajna

Looking at the Exits

UCD Protovis Timeline

Who Escaped?

Fraunhöfer Institute

Results Overview

VRVis-ComVis

Unique Submission Delivery

Victoria University of Wellington

Evacuation Traces Awards

Fraunhöfer InstituteTool Integration

Southern Illinois University EdwardsvilleInnovative Trace Visualization

University of Bari User Testing to Obtain Consensus

Users were asked to view a visualization and “place” a bomb in the right place

The majority of the users (53.84%) marked a point inside the room where suspects 18 & 50 were.

Users speculated that the main suspect either “threw”the device inside that room or activated it upon entering that room.

Wiki Challenge

Wiki logsDataset

Wiki edit comment data Who / what type of edit performed

Analytic SituationThe Paraiso Wikipedia entry became a focal point for conflict between the followers and the general population. The wiki edits pages showed some of the social relationships and dynamics involved.

Wiki QuestionsWhat are the factions represented in the edit pages and who are its members?Is the Paraiso movement involved in violent activities?

Visualizing Edit ActivityPatterns of Edits Over Time

Used a database table and applied various filters to find frequent editors and also color coded entries based on identification of who edits who.

Pacific Northwest National Laboratory

Visualizing Edit ActivityPatterns of Edits Over Time

National Center for Supercomputing Applications at Urbana-Champaign

Palantir

Visualizing Edit ActivityPatterns of Vandalism

U. Washington - PARVAC

Visualizing Edit ActivityPatterns Based on File Changes

File changes over 20,000 bytes stand out showing a deletion and restoration of the entire record. This pattern suggested instances of vandalism where corrections are made by bots

A close view of File Changes under 20,000 bytes reveals patterns indicating editing wars; subsequent offsetting changes in file size by alternating editors.

Oculus Info, Inc.

Visualizing Edit ActivityPatterns of Usage

Carnegie Mellon UniversityPalantir Technologies

Visualizing Individual Supporters/Opponents Revert and Undo Patterns

Vision Systems & Technology, Inc

NEVAC – Penn State University and Drexel University

Visualizing Individual Supporters /Opponents Similarity of Contributors

Oculus Info, Inc.

Palantir Technologies

U. Washington PARVAC

Visualizing FactionsSocial Networks

Inria

Beijing University of Posts and Telecommunications

Visualizing Factions

Vision Systems & Technology, Inc.Oculus Info, Inc.

Wiki Edits Awards

There were none made as outstanding performance in the wiki edits were subsumed in the Grand Challenge awards

Grand ChallengeBased on ALL the data available (i.e. using the data from all 4 Mini Challenges) what is the social network of the Paraiso movement at the end of the time period?What name or names can be associated with individual activities?What is the geographical range of the Paraiso Movement and how does it change over time?How do the major beliefs of the Paraiso movement affect their activities?

Data IntegrationDynamic ontology allows disparate data to be combined and searched

Can build structured data from unstructured text

Metadata is stored with entries (can be edited and shared)


History mechanism available. Using the metadata and the history mechanism, analyst can generate a Powerpoint presentation

Data Integration

Association Graph

NEVAC

Integration of Analysis

Analysts do separate analysis of minis

Share folders which contain sandbox analysis

Another analyst can pick this up and combine in another sandbox

Oculus Info, Inc

Analysis Support

Hypotheses can be added as nodes and links into graphs

NEVAC

Grand Challenge Awards

Oculus Info Inc.Support for Diverse Analytic Techniques

Palantir TechnologiesInteractive Visual Analytic Environment

Pennsylvania State University-NEVACData Integration

AFTER THE BREAK

PanelInteractive session overview3 Grand Challenge teams presentQuestions to teamsOpen discussionNext year plans

VAST 2008 ChallengeThe VAST 2008 Challenge Co-chairs

Georges Grinstein, University of Massachusetts LowellCatherine Plaisant, University of MarylandJean Scholtz, Pacific Northwest National Laboratory

The VAST 2008 Challenge CommitteeSharon Laskowski, National Institute of Standards and TechnologyTheresa O’Connell, National Institute of Standards and TechnologyMark Whiting, Pacific Northwest National Laboratory

Summary of overall event

See slides 2 to 13

Challenge EmphasisThe process and visualizations used to locate the “answers” are AS IMPORTANT as the answers

This is the concept that distinguishes the VAST Challenge from other contestsTherefore we are judging not just the answer but the utility of the visual environment in arriving at that answer.

Judging the EntriesJudges are professional analysts and the VAST Challenge committee Entries are reviewed and both qualitative and quantitative measures are collected

Accuracy scores are also given As the problems and data become more realistic the “answers” are not black and white

Answers need to be supported by evidence

SUNDAY eveningParticipant Session

Opened to Challenge participants only

For each mini-challengeSummarized entriesDiscuss solutions and problemsSuggestions

Monday EveningInteractive Session

3 teams participated 30 minutes to load and preprocess datasets similar to the ones used in the VAST Challenge30 minutes to train analyst, explain system concept and functionalities2 hours for team + analyst to analyze dataObservers

Took notes to provide each team with feedbackGeneral discussion of analysis session

PanelistsDon Pellegrino, Drexel University (NEVAC)Lynn Chien, Oculus Info, Inc.Brian Lee, Palantir

10 minutes each to explain their systemSeveral minutes for questions to each panelistPanelists respond to committee questionsOpen question time

NEVAC– Penn State

Oculus Info, Inc.

Questions- Grand ChallengeWhat was your motivation for entering?What was the most difficult aspect of the Grand Challenge?What was the most rewarding aspect?Can you share with us lessons learned during your work on the Grand Challenge?Would you say that entering the Grand Challenge was worth the effort?

Questions – Interactive SessionAbove and beyond the lessons learned in the Grand Challenge were there additional lessons learned in the interactive session?What was the most difficult aspect of the interactive session?What was the most enjoyable aspect?If a team contacts you for advice on entering the VAST 2009 Challenge what would you tell them?

Audience Questions

Data Set Availability?2008 VAST data sets/questions

Available to all (still on website)If you submit your answers, we will give you the solution

2008 Interactive session data sets/questionsA smaller problemAvailable only to instructors (for training, class projects)2006 and 2007 datasets also availableSolutions and videos given to instructors

VAST 2009 ChallengeVAST 2009 Challenge - Similar formatwww.cs.umd.edu/hcil/VASTchallenge09Practice using previous datasets

Deadlines for submission (June/July)Target dataset delivery in Spring (March?)

Contact any of us if you have suggestions or [email protected]

vast 2008 challenge - university of maryland€¦ · 26/10/2008 · on day 7, this group dropped...

Documents