using log file analysis to evaluate instructional design
Post on 05-Dec-2014
409 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using Log File Analysis toEvaluate Instructional Design
Kenneth W. Fansler
Rodney P. Riegle
Illinois State University04/10/23
Using Log File Analysis toEvaluate Instructional Design
What is a Web server log file? Records activity on a Web server “Who” visited What files were requested What time/date site was visited What page was visited first, last Not “blogs” (Web logs)
Using Log File Analysis toEvaluate Instructional Design
Elements of a log file How to access log files How to generate log file reports Log file analysis software prices and features The limits of log file analysis The ethics of log file analysis How to avoid unsound inferences from log files Toward a Model of Online Instructional Design
Analytics
Using Log File Analysis toEvaluate Instructional Design
Elements of a log file Simple text file Generated by
Web serversoftware
Tracks visitors’activity on aWeb site
Example Difficult to read
in raw format Controlled by
server admin.
Using Log File Analysis toEvaluate Instructional Design
How to access log files
Contact Web server administrator Files are VERY LARGE
Burn to CD or DVD network download
Ask for range of files by date (semester, year, etc.)
Using Log File Analysis toEvaluate Instructional Design
How to generate log file reports Use commercial
reporting softwarelike WebTrends®
Configure thesoftware to reportonly what you wantto see
Ease of use andexpense varies
Example
Using Log File Analysis toEvaluate Instructional Design
Log file analysis software prices and features Variety of software packages All perform same relative
basic function Some provide advanced
reporting functionality All rely on log files Price ranges from
free to >$500 Product list in paper
Using Log File Analysis toEvaluate Instructional Design
The limits of log file analysis
Difficult if not impossible to track usage by “real” person
IP address recorded Limited data recorded on authenticated visitors
Log files record only “request” transactions (client requests page from server)
No Flash No Java apps No cgi applications
Using Log File Analysis toEvaluate Instructional Design
The limits of log file analysis
Log file analysis software usually designed with business, not education, in mind
Results shown in static images (no motion) Easy to misinterpret the results Not useful to students’ self-evaluation
Using Log File Analysis toEvaluate Instructional Design
The ethics of log file analysis
Privacy Any identifiers must be kept confidential
Unsound Inferences Reporting information without revealing limits Tempting to report data as shown in analysis software
Using Log File Analysis toEvaluate Instructional Design
How to avoid unsound inferences from log files
Log file analysis can be misleading
Examples of possible misconceptions based on data analysis
1. Site and page hits are an indicator of the number of visits Different from “page hits” Includes multimedia files related to that html page
Using Log File Analysis toEvaluate Instructional Design
How to avoid unsound inferences from log files
Examples of possible misconceptions based on data
2. All page or site hits are generated by students Search engine “spiders” or “robots” scour web Course authors’ visits are recorded Outside guests who may simply be browsing
3. Hits from a single Internet Protocol (IP) address reflect visits from a single user
Multiple users sharing a single computer One user moving from machine to machine Re-routing of IP addresses through ISPs Firewalls
Using Log File Analysis toEvaluate Instructional Design
How to avoid unsound inferences from log files
Examples of possible misconceptions based on data
4. The “average time viewed” statistic accurately measures the amount of time someone spent on the Web page
“Walk-aways” Multiple browser windows Printers Timeouts
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics Log file analysis currently driven by commercial
interests Customer behavior Raw traffic Purchasing statistics
Online student behavior more difficult to analyze Did learning occur? Engaged interactivity between student and instructor (email,
phone, etc.) Engaged interactivity among students (chats, discussions) Is “Student X” actually “Student X”?
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics
Results Skepticism towards use of log files to analyze online
instruction Continued debate over general effectiveness of online
instruction Problems appear insurmountable to non-experts Inhibited research and theory in regard to using log file
analysis to evaluate online courses If problems can be solved, it will have a huge impact on
instructional evaluation and design
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics
What’s needed? Identify what is currently possible Identify missing characteristics that could improve data
collection Create theoretical model of new tool(s) Work with commercial software designers to integrate
needed characteristics into new software
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics
Characteristics necessary to improve data collection:
1. Log file data must be displayed in 4 dimensions 3 spatial dimensions plus time Replace static charts with motion charts (movies)
2. File data must be tied to individual students More accurate tracking of student activity Navigation tied to final grades, evaluations
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics
Characteristics necessary to improve data collection:
3. Log file data must be integrated with more file types Flash, java applets, cgi programs, email Student navigation can be altered by a single email or discussion
4. Information generated must be accessible in real time by students
Useful to students who want to improve their performance Compare to other students’ behavior
Using Log File Analysis toEvaluate Instructional Design
Toward a Model of Online Instructional Design Analytics Questions still left to ask to support investigation
of new characteristics
1. Do different Web server software applications (IIS, Apache, etc.) log activity differently?
2. Do courseware packages (WebCT, eCollege, Blackboard, etc.) produce a secondary log that more closely tracks individual users’ activity?
3. Will newer versions of Web server software and Web client software work more closely together to provide more detailed logs (such as single-user activity)?
Using Log File Analysis toEvaluate Instructional Design
Questions, Suggestions
Ken Fansler (KenFansler@ilstu.edu) Rod Riegle (rpriegle@ilstu.edu)
top related