1 © 2003 cisco systems, inc. all rights reserved. ptc 6/18/04 what cisco’s adi group is doing in...

1© 2003 Cisco Systems, Inc. All rights reserved.PTC 6/18/04

What Cisco’s ADI Group is Doing in Performance Testing

PTC: June 18, 2004


Agenda

• Organizational Context

• Background of our Organization

• Assessment Strategy and Implementation

• Four Process Delivery Model

• Illustration: NetPASS

• Evaluation

• Moving Forward


Organizational Context


Areas we bring to bear:

• Statistical science

• Computing science

• Assessment science

• Psychological science

• Networking domain knowledge


Some background on our work in the Networking Academies


Global E-Learning Laboratory

Over 450,000 StudentsEnrolled

30,000 to 40,000 Online Tests DailyOver 30 million Total Tests Taken

10,000 + Academies,150 Countries

Over 24,000 Instructors


And there is probably one in your neighborhood….


New Courses Sponsored by IT Industry Leaders:


So we have some decent delivery volume…….

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1,100,000

1,200,000

January

Febru

ary

Marc

h

April

May

June

July

August

Septe

mber

Octo

ber

Novem

ber

Decem

ber

1999

2000

2001

2002

2003


Assessment strategy and implementation in ADI


Some words about assessment implementation

• Assessment is about making inferences about knowledge, skills and abilities from limited data.

• Assessment development is not only about content, but about the models that allow us to make inferences from the observations.


Some over arching thoughts

• We conceptualize and work in assessment from end to end

• We have tremendous pressure to reduce time to market

• We will continually innovate in presentation, scoring and statistical technologies.

• All form assembly approaches require lots of items

• Items need to be complex to fight rote memorization

This suggests assessment and task design and construction are strongest leverage points


Performance Tasks

• Need similarity to real-world tasks

Cognitive

Visual

Physical

• Need to represent the relevant cognitive representations and features


Our primary framework for thinking about Assessment is Evidence Centered Design

• Provides a language for modern assessment

• Suggests structure for accomplishing the assessment

• Supports transfer and sharing


The world is complex ……..


From Mislevy, Steinberg & Almond (in press):

So is Evidence Centered Design

How do we describe the world and the important claims and tasks in the world

What components do the assessment tasks need?

How do we score the responses from tasks?

How do we pass that information to statistical analysis?

How do we understand and report these inferences?


Performance Task Design

• The need and design of performance tasks is determined during the claims and evidence process

What claims are we making about a student at the end of this course? (claims)

What evidence will we look for that determines the degree to which a student has met that claim? (evidence)

What situation can we set up for an examinee to demonstrate this evidence? (task specification)


Our goal is

• Not just to make a lot of content

• But to specify the underlying logic and design so many people can build off that specification

• Knowledge engineering of the domain

• This means we may need to track data at a more detailed level (e.g. field test items).


Claims database for CCNA


Contextualize the claim with scope specification and instructional guidance


Types of evidence you would look for


Representations to the student, and sources of variation


Representations from the student (work products), and features to detect


Add some additional information about relevant cognitive processes


Templates


Task Design

• Domain

A link to the claim or objective to which the task is seeking to provide evidence (e.g. Trouble shoot a 5 router topology running RIP with faults of type …..).

Enumeration of the relevant representations and artifacts (typology, routers, diagram, scenario)

Features to vary in setting (# of routers, # of hosts, type of fault)

Relevant requirement dimensions (collect information, determine problem, apply fix, check, repeat) versus (apply fix to problem that is described).


Task Design Template – Census of Relevant Task Characteristics

• Setting Corporation

Conference Center

University

• Building Length Less than 100m

More than 100m

• Ethernet Standard 10BaseT

100BaseT

• Subgroup Name Teacher

Student

Customer

• Bandwidth for a Subgroup Drop 10Mbps

100Mbps

• Growth Requirements Given

NA


Task Specification Template -Choosing Key Features

• Setting Corporation

Conference Center

University

• Building Length Less than 100m

More than 100m

• Ethernet Standard 10BaseT

100BaseT

• Subgroup Name Teacher

Student

Customer

• Bandwidth for a Subgroup Drop 10Mbps

100Mbps

• Growth Requirements Given

NA


Use Selected Key Task Features to Make Display Model for the Scenario

Problem Statement:

1. Setting sentence: A(n) setting is [create something that is a typical activity for this setting].

2. Building size sentence: The setting is buildingLength long.

3. Network type sentence: The setting has been asked to install a Ethernet Standard network for this [the typical activity for this setting created above].

4. subgroup 1 sentence: The subgroup 1 connections require a bandwidth of bandwidthForASubgroup 1.



7. Force closets sentence: No networking equipment can be stored in the subgroups 1, 2, & 3 area.

8. Location of POP sentence: The link to the internet is located locationOfExternalConnection(POP).


3. Instantiate individual values into the template:

• Simple (LAN Design)

• A small university needs to set up a network for the English department. The English Building is 50 m long. The building needs a 100BaseT network for the English department. Each faculty member’s drop requires a bandwidth of 100Mbps while the student drops require a bandwidth of 10Mbps. The faculty and student areas are housed at opposite corners of the building. The LAN needs to have following additional characteristics:

• 1. The faculty area needs 8 network drops.

• 2. The student area needs 1 network drop.

• 3. One networking device should be placed in the student area. Use the device that would meet the minimum standards in the student area.

• 4. Students need access to 10 network connections to connect laptop computers.

• 5. No networking equipment can be stored in the faculty or student areas.

• 6. The link to the internet is located in the faculty area.


Advantages: A model that…..

• Guides task and test construction

• Provides extremely high efficiency and scalability

• Forms the basis of defensible evidence for high stakes tests

• By relating task characteristics to difficulty, we can create tasks with known properties in advance!


And…

• These processes are being applied to simulation features as well


Mini-lecture:The Four Process Model for Assessment Delivery

A language for modern assessment implementation


The Four Process Delivery Model

Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

ActivitySelection Process Presentation Process

Task/Evidence

CompositeLibrary

EvidenceIdentification

Process

EvidenceAccumulation

Process

AdministratorExaminee


Activity selection determines the activities to be invoked in the presentation process



Task/Evidence

CompositeLibrary


Process


Process

Messages



Evidence identification looks for important features of the work product (can be used for feedback).



Task/Evidence

CompositeLibrary


Process


Process

Work Product

Messages



Evidence identification looks for important features of the work product (can be used for feedback).



Task/Evidence

CompositeLibrary


Process


Process

Work Product

Messages


Task Level Feedback


The result of evidence identification is values of observations



Task/Evidence

CompositeLibrary


Process


Process

Observations

Work Product

Messages


Task Level Feedback


Evidence accumulation synthesizes the information for summary feedback



Task/Evidence

CompositeLibrary


Process


Process

Observations

Work Product

Messages


Summary FeedbackTask Level Feedback


In adaptive situations, evidence accumulation informs activity selection



Task/Evidence

CompositeLibrary


Process


Process

ExamineeRecord

Observations

Work Product

Messages




Some other processes go on in the background



Task/Evidence

CompositeLibrary


Process


Process

ExamineeRecord

Observations

Work Product

Messages

PresentationMaterial

Evidence RuleData

Weights ofEvidence

Description DataAdministrator

Examinee


Find Task Fetch Task

Fetch Scoring DataFetch Weights


Four Process Model and Performance Tasks

• Performance tasks are really just a different means of presenting tasks to examinees


Evidence Identification of Performance Tasks

• Performance Tasks can be scored just like other more traditional tasks

0/1 (all or nothing)

Partial credit (degrees of correctness)

Different observables providing evidence for different student model variables


Evidence Accumulation and Performance Tasks

• Assessments containing performance tasks can still be summarized the same as more traditional assessments

Add number of correct

Ability estimates based on IRT

Probability estimates based on Bayesian Inference Nets


Activity Selection and Performance Tasks

• Assessments containing performance tasks can still implement the same activity selection procedures as do the more traditional assessments

Go to the next task in a specified sequence

Randomly select a task from a specified bucket

Task selected by examinee

Task chosen through adaptive algorithm


Illustration: NetPass Research Prototype

• Create on-line performance assessment of networking skills

• Focus on learner feedback rather than high-stakes testing

• Learn something about assessment in general


The task starts with a scenario and description of goals


To capture their mental model of the network, we ask them to draw the network with a diagramming tool


The diagram is created by dragging and dropping icons


When more detailed information is required, it is collected as well


When students are done, they press “Submit”…


And the graphical representation is converted into a text representation


The text file is scored following detailed rules, which result in characterizations of the work


And feedback is created for the student…


New Computational Tools Evolving all the time: A Bayesian Inference Network

Design

Network modeling

Troubleshoot Implement (Configure)

Domain disciplinary knowledge

Network Proficiency

Media

Addressing

OSI

Hardware

Security

Protocols

IOS

Correlation

Part-of

Part-of

Part-of

Part-of

Part-ofPart-of

Part-of

Part-of

Part-ofPart-of

Prequisite

Correlation

OperationPlanning

Part-of

Part-of

OSI


Summary Feedback – Probability of Mastery


Performance Tasks can also be Multiple-Choice


Assessment Evaluation


Exam Analyses


Examinee Comments


Item Performance


Moving Forward


Lessons Learned: Scoring Performance Tasks

• How can we get the most information from these performance tasks?

• Scoring the Product

–What if there is more than one accepted outcome?

• Scoring the Process

–More than one accepted process?

–Is there really a ‘right’ way?

• Be Wary of Expert Judgment

–Need to gather data to verify


Future Directions

• Figuring out how to present the performance task is only one aspect to delivering a performance assessment

• The issues is really applying existing technologies and processes to more complex performance-based items and assessments

1 © 2003 cisco systems, inc. all rights reserved. ptc 6/18/04 what cisco’s adi group is doing in...

Documents