you’ve collected it … now what ? exploratory data analysis for not-very-big data
TRANSCRIPT
Telling Stories with your data
Graphs, Tables and Basic, Basic Statistics with SAS Enterprise
Guide ®
You’ve collected it …NOW WHAT ?
Exploratory data analysis for
not-very-big data
Our sampleData from the pilot study for Spirit Lake: The Game, an educational game for students in grades four through six
Our Enterprise Guide Project
A first look at the data, Filter data setsTables of descriptive statistics
Our Enterprise Guide Project
A second look
Cross-tabulations, Graphics, Summary tables T-test
Let’s replace math class with a game ……. Incredibly, the schools went along with this!
Our sites Two intervention schools, six classrooms
Three fourth-grade classes
Fifteen students (five each) from three fifth-grade classes
One control group school with one fourth-grade and one fifth-grade
All on the same American Indian reservation
Exercise 1: ready
Figure 1.1
FILE> OPEN> DATA
Figure 1.2
Tools> options
Figure 1.3
Results General> RTF under Result Formats
Figure 1.4
Great. You have data and you are set to have pretty results.
What now?
Tasks> describe> characterize data
Figure 1.5
ALWAYS DO THIS !!
Just click through the windows and accept all of the defaults.
Especially this one (you’ll find out why shortly)
Real grown-up statisticians know ….
Look for errors in data
Look for missing data
There is no teacher named “test”
This outlier with the perfect score wasn’t a student, it was the answer key!
Filtering data
Exercise 2: set
Figure 2.1
OR……
Tasks > data > filter and sort
Figure 2.2
Double arrows select all of the variables.
Figure 2.3
Filter out records
Click on the …
Exercise 3: go!
Is 64% missing okay?
Exercise 3: go!
Variable Label N NMiss Total Min MeanMedian Max StdMean
Gender Gender 67 21 99 1 1.4776 1.0 2 0.06148
Variable Label N NMiss Total Min MeanMedia
nMa
x StdMean
Grade grade 83 5 362 4 4.314 4.0 5 0.05305
Table 3.1
Table 3.2
DATA
We naively believed that there would be little attrition over a 10-
week period
"He's just gone".
Lesson learned Real data analysis is
more than pointing and clicking, it’s learning to ask questions from the results you get and learn from the answers to those questions
Data visualization at its most basic
Figure 3.1 Figure 3.2
Take Note!
File > new note
Figure 3.3
Figure 3.4
Right-click on note to link to procedure that inspired your note
Figure 3.4
Exercise 4: table analysis- looking a little closer
TASKS> DESCRIBE> TABLE ANALYSIS
Figure 4.1
CHANGING THE DATA SET
PULL DOWN TO SELECT DATA SET
Drag the variables, School and Grade, under the Table variables heading.
Drag “School” to top of the table as your column variable and “grade” to the side as your row variable.
Table of grade by School
grade(grade) School(School)FrequencyCol Pct
CONTROL EXPERIME Total4 15
55.5638
67.8653
5 1244.44
1832.14
30
Total 27 56 83
Frequency Missing = 5
What of the students with complete data
Is there still a disproportion by grade?
Right-click on the Table Analysis icon
Drag the variable “missdata” under Group Analysis By
Table of grade by School
grade(grade) School(School)FrequencyCol Pct
CONTROL EXPERIME Total4 4
80.0010
62.5014
5 120.00
637.50
7
Total 5 16 21
Frequency Missing = 5
TOO MUCH MISSING DATA MAKES PEOPLE GRUMPY
WHO ARE THESE PEOPLE MISSING DATA?
Figure 4.9
What Happened? To insure anonymity and protect student data, we
never had the students' names – the teachers had a roster of students matched with username. The experimental school was able to fill in the blanks for all students missing grade data, the control group school did not.
What We Did About It This isn't something we just waved our hands about
and moved on. It seriously concerned us. The degree of missing data overall disturbed us, as did the fact that we didn't seem to be able to get follow-up data. It concerned us enough that we hired a data coordinator on each reservation where we are testing in the upcoming year and will be analyzing the pretest data as it comes in and trying to update any missing data we can in the same week it is collected. This is the purpose of pilot studies, to find problems and fix them.
What is an item analysis and how is it helpful?
Two types of item analysis.
Examination of the distribution of responses which choice the student selected “a”,”b”, “c” or “d” as the correct answer. (Look at CHARACTERIZE DATA plots)
Does one of the distractors gets selected more often than the correct answer?
Item difficulty analysis
Examine what percentage of students answered each item correctly
A basic means of establishing test validity. One would expect that items at the second-grade level would have the lowest level of difficulty, items at fifth-grade level the highest difficulty, and be answered correctly by the fewest students.
If items are scored 0 = wrong, 1 = right, can use the means to see what percentage of students answered correctly.
EXERCISE 5: ITEM ANALYSIS TWO WAYS WITH THE CHARACTERIZE DATA
TASK
Item Difficulty Analysis in Six (or fewer) Easy Steps
1. Click on the univariate statistics data set produced by the CHARACTERIZE DATA TASK to select it
2. From the top menu, select TASKS > DESCRIBE > LIST DATA
3. From the Variables to assign pane, select the ones you want in your report, in this case Variable, N, NMISS, Mean, Min and Max.
EDIT
4. Select the records you want in your report.
5. Format the columns in the report
Figure 5.4
I click the CHANGE button next to format.
Then I click on Numeric for the format category. >
6. Next, click Options and un-check the box next to Row Numbers
Figure 5.7
Item N NMiss Mean Min Max
postsc2 68 20 0.88 0 1
postsc3 68 20 0.87 0 1
postsc4 68 20 0.81 0 1
sc3 82 6 0.8 0 1
sc2 82 6 0.78 0 1
sc4 82 6 0.78 0 1
postsc18 68 20 0.74 0 1
postsc8 68 20 0.68 0 1
postsc1 68 20 0.65 0 1
COPIED AND PASTED INTO EXCEL & SORTED
What does this table tell you?
What data tells us Since items are in order of grade level, first few
items should be answered correctly by the most people.
Expected pattern holds for post-test and pre-test, although it's not perfect
A higher percentage of students answered the post-test questions correctly than the pretest, as we would hope
Unexpectedly, items 5 and 6 have some of the lowest percentage correct of any item
EXERCISE 6: GRAPHING ITEM DIFFICULTY
(If you forget the Sum variable, you'll just get a chart that shows each item occurred in the data set once. Not very helpful.)
SELECT ONLY THE ITEMS OF INTEREST
Click Layout tab under Appearance and select Descending Bar Height
Figure 6.5
Item Difficulty AnalysisPost-Test
Compare with pre-test data
Right-click on the bar chart icon in process flow and select Modify.
Three modifications are needed: Click on the EDIT button and change the
filter. First click the X at the end of the row to delete the current filter. Then select “item” and “in a list” and the variables sc1 – sc24 for your items to chart.
Change the chart title.
X axis must be the same, from 0 to 1 so you can compare the two charts
How to set the X axis. Click on Major Ticks and then under Major Horizontal
Ticks click Specify. In the input box on the top right, enter each of the major ticks you want (from 0, .2 to 1) and click ADD.
Pre-test
Post-test
Our story so far …
We've found one error, that the answer key was left in as a record.
We've seen that we have an issue with missing data that needs to be fixed
It appears that the test is reasonably reliable, although, of course, more sophisticated statistics are needed to examine that issue further.
We've also realized that we can't really compare the pretest and post-test since we have a large proportion of missing subjects. We need to match pre- and post-test scores.
EXERCISE 7: GETTING DOWN TO BUSINESS WITH T-TESTS
TASKS > ANOVA > T-test
Control GroupResults
Table 7.1
Experimental Group
Results
The moral of the story Experimental group improved more
Students who played the game less improved less
Differences were not explained by outliers in either group
There was a definite shift in the distribution of the experimental group only
CONCLUSION Exploratory data analysis is a key first step
A few simple tasks in SAS Enterprise Guide can go a surprisingly long way
While exploring your data, it’s crucial to note the concerns raised, follow-up questions and policy recommendations that come out of your analysis
Thank YouU.S. Department of
Agriculture
Small Business Innovation Research
Rural Economic Development
Teachers and students of the
Spirit Lake Dakota Nation
ContactThe Julia Group/ 7 Generation Games
2111 7th St. #8
Santa Monica, CA 90405
(310) 717-9089
http://www.thejuliagroup.comwww.7generationgames.com