Selection of irrelevant navigation paths during digital reading:
Underlying cognitive processes
Ladislao Salmerón
ERI Lectura, University of Valencia, Spain
ProcessDataConf2019
Overview
• Navigation paths
– Indicators
– Interpretation
– Context dependencies
• Study
• Conclusions
ProcessDataConf2019
Navigation paths
ProcessDataConf2019
· Major difference between print and digital reading is the presence of hyperlinks that create non-sequential page structures (OECD, 2011, 2015)
· Strong association between quality of students’ navigation and digital reading …
… at country (OECD, 2015) and at individual (e.g.
Naumann & Salmerón, 2016) level. … even after accounting for performance in print reading.
In digital reading, process data is already (partially) accounted in performance scores.
Navigation paths
ProcessDataConf2019
(OECD, 2015)
Navigation paths
ProcessDataConf2019
· “Savvy digital readers are expected to navigate across pages that contain information relevant for the question to be answered, or the task to be solved”
· Process indicators of navigation quality (Naumann, 2015)
1. Number of relevant pages visited2. Total number of pages visited3. Derived indicators (% relevant pages)
1
2 3
4
A page is Relevant if informationfor the response must be:-located in p. 4-related p. 1 to p.4 -related 1-2-3-4
Navigation paths
ProcessDataConf2019
A page is irrelevant if itdoesn’t containinformation useful to answer the question, or isnot in the path guiding to a relevant one
Navigation paths
ProcessDataConf2019
· Cognitive processes underlying navigation:
a) To relevant pages > inferential process
· Print reading comprehension (Hahnel, Goldhammer, Kröhne,
& Naumann, 2017, 2018; Naumann, Richter, Christmann, & Groeben, 2008; Naumann & Salmerón, 2016; OECD, 2011, 2015; Salmerón & García, 2011)
· Progression across high-school years (Keil & Kominsky,
2013; Salmerón, García & Vidal-Abarca, 2018)
Navigation paths
ProcessDataConf2019
· Cognitive processes underlying navigation:
b) To irrelevant pages:· Generally interpreted as ‘Lostness’ (Shapiro &
Niederhauser, 2004)
· Its isolated effect has been less studied in the past
Navigation paths
ProcessDataConf2019
· Scenarios in the Digital reading assessment in PISA
include highly differentiated sections.
Navigation paths
ProcessDataConf2019
· Can visits to irrelevant pages also represent students’
attempts to capture a complete view of the theme?
- Self-regulated vs. cue dependent navigation
(Balcytiene, 1999)
· This aspect should be more obvious in digital texts that
are highly topically interconnected, such as Wikipedia
texts, than in scenarios with highly differentiated sections,
such as web portals.
Goal
ProcessDataConf2019
· Test if visits to irrelevant pages differ across digital
contexts (less / more interconnected):
-Relationship with print reading skills
-Relationship with digital reading scores
-Potential mediation between the former
Study
ProcessDataConf2019
Participants555 students (12-16 years old) · 5 schools (2 rural, 3 urban)· 40 intact classes
Exclusion criteria: learning disability, curricular adaptation
In class participation. Two sessions of 50’
Grade 7th 1548th 1419th 14310th 137
ProcessDataConf2019
• Students’ print comprehension skills (reading literacy) were assessed using a standardized paper and pencil test (Llorens et al., 2011).
• Texts: – a textbook excerpt about global warming, – an article from a popular science magazine about the language
of the bees, – a pamphlet about the postural ergonomics of work chairs, – two letters to the editor about the pros and cons of nuclear
energy,– a newspaper diagram about traffic accidents.
• Each unit included 3-5 questions covering the different aspects suggested by the PISA framework
• Cronbach’s alpha = .80.
Method
Method
ProcessDataConf2019
· Two web reading scenarios were designed following thePISA (OECD, 2009) scheme.· Participants were given one question at a time, and weretold to use the information from the web pages provided to answer the question.· Questions were constructed following three aspects(OECD, 2009): access and retrieve, integrate and interprete, and reflect and evaluate.· Heterogeneous relationship between question and relevant hyperlinks: word match, paraphrases, irrelevant word matches (Cerdán, Gilabert, & Vidal-Abarca, 2011)
Method
ProcessDataConf2019
· Web portal scenario:-Portal dedicated to young people-Divided into five categories (sports, courses, technology, nature and health), with three subcategories each. -12 questions required that participants clicked on at least two (maximum seven) links to access relevant information to answer the question, and involved both textual and graphic information, such as tables or maps.
Method
#19. You would like to learn how to sing, so you decide to enroll in a course offered by the Youth Portal. Search theaddress for the center. a) St. Del Padre Vitoriab) St. Mayor 25c) St. De Donoso Cortésd) St. Goya
ProcessDataConf2019
Example question from Web portal scenario
ProcessDataConf2019
Method
ProcessDataConf2019
· Wikipedia scenario:-Main document on the topic of the French Revolution -Included 10 subsections and 24 embedded hyperlinks to additional Wikipedia pages. - 6 questions required that participants clicked on at least one (maximum two) embedded hyperlink(s) to access relevant information to answer the question.
Method
#13. Why the policy of the Antient Regime was against the new social ideas from the French Revolution? Because…
a) … contrary to the Ancient Regime, the new society valued only daily lifeproblems.b) …social ideas from the Ancient Regime were based on the power of thearmy while new ideas were based on reason.c)… in the new regime the philosophers governed society and in the oldregime monarchs and the Church ruled.d) … the new social ideas were based on reason, and the old in divine law.
ProcessDataConf2019
Example question from Wikipedia scenario
ProcessDataConf2019
Method
• Log-files indicators:
– Number of unique pages visited
– Relevant and irrelevant to the task
– By scenario (Wikipedia and Web portal)
ProcessDataConf2019
Results
ProcessDataConf2019
Min Max Mean SD Asymmetry
Skewness
Unique Relevantvisits Wikipedia
0 1.67 .37 .33 .41 -.74
Unique Irrelevantvisits Wikipedia
0 .33 .06 .10 1.28 .72
Unique Relevantvisits Web Portal
0 3.33 2.0 .74 -1.17 .63
Unique Irrelevantvisits Web Portal
0 .33 .10 .40 .39 -.59
Descriptive scores for navigation indicators, averaged by question
ProcessDataConf2019
Print readingcomprehension
Relevant visits
Digital readingcomprehension
Irrelevant visits
Results
r = .45*
r = -.15*
r = .53*
r = -.13*
Web portal scenario:Pearson correlations
Results
Web portal scenario
Visits Visits
Pri
nt
Rea
din
g co
mp
reh
ensi
on
Dig
ital
Rea
din
g co
mp
reh
ensi
on
Unique relevant visitsUnique irrelevant visits
ProcessDataConf2019
ProcessDataConf2019
Print readingcomprehension
Relevant visits
Digital readingcomprehension
Irrelevant visits
Results
r = .40*
r = .15*
r = .61*
r = .31*
Wikipedia scenario:Pearson correlations
Results
Wikipedia scenarioUnique relevant visitsUnique irrelevant visits
Visits Visits
Pri
nt
Rea
din
g co
mp
reh
ensi
on
Dig
ital
Rea
din
g co
mp
reh
ensi
on
ProcessDataConf2019
ProcessDataConf2019
Print readingcomprehension
Relevant visits
Digital readingcomprehension
Irrelevant visits
Results
.45*
-.15*
.42*
-.17*
Web portal scenario:Mediation analysis (Bootstraping, Preacher & Hayes (2008))
c’= .34*
Indirect effects:-relevant visits: .18 [.14, .24]-irrelevant visits: .03 [.01, .05]
R2adj = .43*
ProcessDataConf2019
Print readingcomprehension
Relevant visits
Digital readingcomprehension
Irrelevant visits
Results
.40*
.15*
.56*
-.02, n.s.
Wikipedia scenario:Mediation analysis (Bootstraping, Preacher & Hayes (2008))
c’= .22*
Indirect effects:-relevant visits: .22 [.17, .27]-irrelevant visits: -.01 [-.02, .01]
R2adj = .44*
Conclusions
• Irrelevant visits in the Web portal scenario, with highly differenciated topic structure:
– Negative correlations with print readingcomprehension and digital reading performance
– Indirect effect of print reading comprehension to digital reading performance.
ProcessDataConf2019
Conclusions
• Irrelevant visits in the Wikipedia scenario, with highly interconnected topic structure:
– Positive correlations with print readingcomprehension and digital reading performance
– It adds no variance to digital readingperformance.
ProcessDataConf2019
Conclusions
• Underlying cognitive processes?
– In loosely interconnected texts: Distraction?(must be independent of inferential processes, accounted for by relevant visits effects).
– In highly interconnected texts: Byproduct of inferential processes (exploration, curiosity, Balcytiene, 1999)
• Need to complement log-files withqualitative data (verbal protocols, interviews)
ProcessDataConf2019
Conclusions
• Practical implications:
– Consider ‘irrelevant visits’ with caution when building indices of ‘navigation quality’.
– Navigation indices may be dependent on the interconnectedness of the scenarios
– ‘Unique relevant visits’ alone are a strong predictor of performance across scenarios
ProcessDataConf2019
Thanks for your attention
www.uv.es/lectura
www.uv.es/lasalgon
Twitter: LSalmeron_UV
ProcessDataConf2019
Unique relevant visitsUnique irrelevant visits
Materials
• Q: at your school garden your class is cultivating some plants on danger. Do you think you could cultivate a ‘Clover water’?
Spanish plants on danger
One of these [extingueshedplants] is the ‘Clover water’. Scientists have found spores of it, which allows its use in the lab and its reintroduction in nature.
Materials
• Q: The Youth Portal just posted a report on the extinction danger of mammals. What is the extinction level in Canada?
Animals in danger of extinction across the planet
Extinction risk variesaccording to the continent