A USEABLE, ONLINE NASA-TLX TOOL

David Sharek Psychology Department, North Carolina State University, Raleigh, NC 27695-7650 USA

For over 20 years, the NASA Task Load indeX (NASA-TLX) (Hart & Staveland, 1988) has

been successfully used as a self-report measure of cognitive workload, yet emphasis on the

usability and accessibility of the NASA-TLX as a research tool has remained lacking. A freely-

available, user-friendly, online version of NASA-TLX is introduced (Sharek, 2009). Key

features and benefits of the online version for researchers and participants are discussed.

INTRODUCTION

The NASA-TLX is a subjective,

multidimensional assessment tool that can be used

to rate perceived workload on six subscales: Mental

Demand, Physical Demand, Temporal Demand,

Performance, Effort, and Frustration. It was

developed by the Human Performance Group at

NASA Ames Research Center over a three year

development cycle that included more than 40

laboratory simulations (NASA, 1986). It has been

cited in over 550 studies (Hart, 2006) and a recent

search for “NASA-TLX” on Google Scholar

revealed over 4,820 articles (Google, n.d.). These

statistics highlight the large influence the NASA-

TLX has had in Human Factors research.

Some electronic versions of the NASA-TLX

were designed many years ago (NASA, 2003;

NCARAI, 2004) and have not been updated. One

version is available as part of a software package

that costs over 400 dollars (Inquisit, 2010); the Web

version costs over 1000 dollars. Another version

lacks instructional text for participants (Cao,

Chintamani, Pandya, & Ellis, 2009) and doesn’t

provide definitions for the subscale items during the

pairwise comparison section. All versions require a

download and installation on Windows compatible

computers since Apple or Linux machines are not

supported. These limitations, coupled with often

poorly designed user interfaces have been addressed

in the form of a freely available, Rich Internet

Application (RIA) version of the NASA-TLX

(Online-TLX).

The Online-TLX is not subject to any of the

previously discussed limitations. Additionally,

based on user feedback, new updates with enhanced

features are released regularly. A login system

allows researchers to store all their data in one

location so they can concurrently run multiple

instances of the software, for example, in computer

labs, or when conducting completely Web-based

experiments.

ONLINE-TLX TOOL OVERVIEW

The Online-TLX can be accessed at

http://www.NASATLX.com. The URL for the tool

was created to be both easy to remember and

descriptive. The entire file size for the program is

210KB and typically loads in less than one second

on a consumer-grade high speed Internet

connection. Maintaining a small file size was

important during the development process so that

end-user download times would be kept to a

minimum. A MySQL database using PHP as the

server-side scripting language manages the data.

The Online-TLX was developed in Adobe Flash

CS4 using ActionScript 3. The minimum required

Flash player is version 9.0. The decision to develop

the tool in Flash was based on the 98.9%

penetration rate for version 9 of the plug-in (Table

1) in mature markets (Adobe, 2011). Additionally,

the decision to use Flash as the developmental

platform was due to Flash’s cross-platform

compatibility. That is, the intrinsic nature of a Flash

file (.SWF) allows it to accommodate a variety of

operating systems that run on Windows, Apple, or

Cop

yrig

ht 2

011

by H

uman

Fac

tors

and

Erg

onom

ics

Soc

iety

, Inc

. A

ll rig

hts

rese

rved

DO

I 10.

1177

/107

1181

3115

5128

6

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011 1375

Linux machines. This allows for rapid-prototyping

and development and reduces potential formatting

and programming errors across platforms.

Table 1. Flash Player Ubiquity (Adobe, 2011)

Flash Player 9

Mature Markets

US, Canada, UK, Germany, France,

Japan, Australia, New Zealand

99.6%

US/Canada 99.9%

Europe

UK, Germany, France

99.7%

Japan 98.6%

Australia/New Zealand 99.1%

Emerging Markets

China, S. Korea, Russia, India,

Taiwan

98.8%

WALKTHROUGH

Login Screen

After the tool has loaded, the login screen

(Figure 1) is displayed. This screen allows

returning users to enter their user ID and password.

Figure 1. Online NASA-TLX (Sharek, 2009) Login

Screen

New users can register by clicking on the Sign

Up button at the top right of the screen. Clicking on

this button simply loads an additional password text

box and changes the instructional text (Figure 2).

This allows any email or password information that

a user may have already entered to remain, so that

only the secondary password textbox needs to be

filled out. Email addresses are used for user IDs.

This reduces the amount of information required at

sign up, and ensures the selection of a unique user

ID. Additionally, if users forget their password, a

new temporary password can be sent via their email

address. Passwords are stored as secure 32-digit

hexadecimal values using a MD5 cryptographic

hash function.

Figure 2. Online NASA-TLX (Sharek, 2009) Sign up

Screen

Options

After a researcher logs in, they are presented with

an Options screen (Figure 3). From this screen,

researchers can setup and begin a new TLX

measurement session, or they can select to

download all the data that have already been

collected in previous sessions.

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011 1376

Figure 3. Online NASA-TLX Options Screen.

In order to begin a new TLX measurement

session, a Participant ID must be entered. This

value should help the researcher to identify who

was participating in the task. If a participant is

taking part in multiple TLX studies, an optional

Experiment ID value can be entered. The use of

these values is left up to the researcher, but using a

standard convention for identification is

recommended.

Researchers also can choose to include the 15

pairwise comparison questions by clicking on a

radio-type toggle button. According to Hart (Hart &

Staveland, 1988) if participants are required to use

the TLX tool multiple times, they only need to

answer the pairwise comparisons once per task type.

If a participant’s workload needs to be measured for

intrinsically different tasks, then revisiting the

pairwise comparisons may be required. However,

according to one school of thought, removing the

pairwise comparisons altogether may actually

increase experimental validity and reduce

experimental error (Bustamante & Spain, 2008).

All TLX session data are stored in a database

and are always available to be securely downloaded.

This allows the TLX to be administered in computer

lab settings across multiple workstations without the

need to save data locally to each machine; this can

become a laborious task when retrieving the data

from each machine at the end of the experiment.

Data are saved in a comma delineated .CSV file.

This file type can be natively opened by Microsoft

Excel as well as traditional statistical software

packages such as SPSS and SAS. Additionally, data

is formatted so that it can be easily analyzed. That

is, each row of data represents one participant, and

each column represents a variable. A total of 23

variables are saved for each participant including:

researcher ID, experiment ID, participant ID,

overall TLX workload score, raw ratings for each

subscale, individual weights based on the pairwise

comparisons for each subscale, weighted workloads

for each subscale, and timestamp.

The primary benefit for storing as much data as

possible is that researchers have the freedom to

organize and conduct a variety of analyses. A

secondary benefit for including this large amount of

data is that trust in the tool may be increased

through revealing all data used in the calculations.

Each time a file is downloaded, it is saved

using a standard naming convention that includes

the user’s ID and a versioning system which helps

reduce the potential to accidentally overwrite a file.

Currently, all TLX sessions are downloaded to the

file, but future iterations will allow users to only

download data that matches specific participant IDs

or experiment IDs.

Subscale Ratings Screen

After researchers set up the tool and populate

the Participant ID and the optional Experiment ID,

they can click on the Next button to end the setup

process and initiate the start of the TLX. An

introductory screen (Figure 4) is displayed that

includes instructions for participants to read before

they begin. These instructions are editable and can

be modified and saved by the researcher. After

reading the Instructions, participants click on the

start button to begin the ratings.

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011 1377

Figure 4. NASA-TLX Instruction Screen

Figure 5 shows the subscale rating screen. In

this screen, participants are asked to rate their

experience with the task on all six subscales. The

instructions and definitions remain on-screen so that

participants won’t have to memorize the operational

definition for each subscale. As can be seen in

Figure 5, participants also can retrieve an extended

definition for each subscale by hovering over each

definition.

Figure 5. Online NASA-TLX Subscale Ratings

Screen. A hover help is displayed (shown) when a

user hovers over the shortened definition for each

subscale.

Participants can click anywhere along the

continuum for each subscale. The rationale for

providing a continuum was so that the scale would

map as closely to the original paper and pencil

version as possible. If a participant needs to adjust

their rating, they can click on the new point along

the continuum and the high-contrast indicator will

automatically scroll to the new location. The use of

animation here is to provide a visual confirmation

that a change has been made to the scale. If all six

ratings have not been made then a forcing function

will prevent the participant from advancing,

otherwise, clicking on the Submit button will allow

them to continue.

Pairwise Comparison Screen

After completing the subscale ratings, and if the

pairwise comparisons option was selected, the

participant will be presented with a screen similar to

the one in Figure 6. All six subscale definitions

remain on the screen while a smaller window

displays the individual comparisons. Clicking on

the subscale that contributed the most to the task

will highlight that subscale to indicate it has been

selected. The participant then must click on the

Submit button to advance to the next comparison.

This two-step selection process allows the

participant to change their mind. It also prevents a

participant from quickly clicking on the same

button in order to advance to the end of the

measurement without thoughtfully participating.

After a comparison has been submitted, the window

slides off screen to the left while a new comparison

window slides in from the right. This quick

animation indicates to the participant that a new

comparison is ready for their selection. After the

final selection is made, the participant is taken to a

debriefing page. The text for this page can be edited

by the researcher. Additionally, the participant can

be transferred to another Web page of the

researcher’s choosing.

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011 1378

Figure 6. Online NASA-TLX Pairwise Comparison

Screen.

FUTURE DEVELOPMENT

Mobile versions of the Online-TLX are

currently under development. An iPhone/iPod

Touch version will be released first followed by a

version for the Android mobile operating system.

Releasing the Online-TLX to mobile platforms

provides the potential to facilitate easier workload

measurements in the field or in situations where use

of a personal computer or laptop is not possible.

A Feedback button is available on the Login

screen, and all users are encouraged to submit

feedback regarding new features. The further

development of features for the Online-TLX will be

influenced by user input and continued usability

analysis of the tool.

REFERENCES Adobe. (2011). Flash Player Version Penetration, from

http://www.adobe.com/products/player_census/flashp

layer/

Bustamante, E. A., & Spain, R. D. (2008). Measurement

invariance of the NASA TLX. Human Factors and

Ergonomics Society Annual Meeting Proceedings,

52, 1522-1526.

Cao, A., Chintamani, K. K., Pandya, A. K., & Ellis, R. D.

(2009). NASA TLX: Software for assessing

subjective mental workload. Behavior Research

Methods, 41(1), 113-117. doi: 10.3758/brm.41.1.113

Google. (n.d.), Retrieved June 15, 2011, from Google Scholar,

http://scholar.google.com/scholar?&q=%22nasa-

tlx%22

Hart, S. (2006). NASA -Task Load Index (Nasa-TLX); 20

Years Later. Human Factors and Ergonomics Society

Annual Meeting Proceedings, 50, 904-908.

Hart, S., & Staveland, L. (1988). Development of NASA-TLX

(Task Load Index): Results of empirical and

theoretical research. In P. Hancock & N. Meshkati

(Eds.), Human mental workload (pp. 139-183).

Amsterdam: North Holland.

Inquisit. (2010). NASA Task Load Index (NASATLX)

(Version 3.0.5.0) [Computer Software]. Seattle, WA:

Millisecond Software LLC. Retrieved from

http://www.millisecond.com/download/samples/v3/N

ASATLX/default.aspx

NASA. (1986). Nasa Task Load Index (TLX) v. 1.0 Manual

NASA. (2003). NASA Task Load Index (TLX): Computerized

Version (Version 2.0) [Computer Software]. Moffett

Field, CA: NASA-Ames Research Center, Aerospace

Human Factors Research Division. Retrieved from

http://humansystems.arc.nasa.gov/groups/TLX/comp

uter.php

NCARAI. (2004). NASA TLX for Windows (Version 1)

[Computer Software]. Washington DC: Navy Center

for Applied Research in Artificial Intelligence.

Retrieved from

http://www.nrl.navy.mil/aic/ide/NASATLX.php

Sharek, D. (2009). NASA-TLX Online Tool (Version 0.6)

[Internet Application]. Raleigh, NC. Retrieved from

http://www.nasatlx.com

Sharek, D. (2011). NASA-TLX Desktop Tool (Version 2.0.1)

[Computer Software]. Raleigh, NC. Retrieved from

http://www.playgraph.com/apps/nasa-tlx

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011 1379

DESKTOP VERSION A free stand-alone version also is currently

available at

http://hfs.sagepub.com/supplemental

for both Windows and Apple machines (Sharek, 2011). The stand-alone version was developed using Adobe Air’s RIA environment. This development platform essentially allows for the same online version to be deployed to the desktop with only minor modifications thus reducing the development load when new updates or features are released.

http://pro.sagepub.com/supplemental

,