Expanding Eye-Tracking Methods to Explain the Socio-Cognitive Effects of Shared Annotations

Introduction

Annotated textbooks constitute a learning resource that should remain in the history of knowledge forever.

The social vocation of annotations slipped away after the advent of the printing press. Originally the manuscripts were annotated only by relevant academic authors. The produced glosses were shared to facilitate the scholarly need for elucidation and reinterpretation of the obscure passages and terminology of the texts. Nowadays students take notes individually, and engage occasionally in the spontaneous practice of note sharing through informal meetings, email or other forms of electronic communications.

We designed a head-mounted eye tracking system combined with a vision-based system for document image recognition and annotation extraction. We used this system in a control study to intestigate and explain the socio-cognitive effects of textbooks annotations. The obtained results justified the design and further experimentation of a lightweight note sharing system in the classroom.

The results of such experimental journey and the robustness of the technological framework for document image processing resonate throughout the design implications on the new rising platforms of massive open online courses.

A Head-Mounted Eye Tracking System for Studying Natural Reading

A head-mounted eye tracking system was designed and developed through several iterations of ad-hoc tests. Some ergonomic issues and visual tracking adaptations were implicated at the design phase.

We finally used this system in a controlled experiment, in order to study how learners read and reread annotated paper texts. The following video shows the salient computational features of the system.

 

The framework is based on a system for automatic detection, localization and extraction of all the annotations taken on paper.

Through this experiment we measured the phenomenon of annotation-induced overt attention shifts in reading, and found out that readers tend to link a text segment with an annotation when it is in their early peripheral vision. Furthermore, we discovered that the rate of attention shift can predict retained learning.

 

Cognitive and Social Effects of Annotations

In 2011, we carried out a controlled experiment with 57 participants. They were asked to read a text and answer comprehension questions: the control group without taking annotations, the second group reading and taking self annotations, and the third group reading a peer-annotated version of the same text. The results show that (1) the number of self-produced annotations had a high positive correlation with the learning gain, (2) the group that read an annotated version of the text performed significantly better than the control group in terms of learning gain.

The obtained results justified a further experimentation. In 2013 we deployed a lightweight note sharing system in the classroom (exemplified in the figure below).

 

The system was designed for the classroom context and evaluated through a longitudinal study lasting for an academic semester and involving 20 participants, enrolled in a Master-level course in computer science. Three key findings emerged. First, the tool spontaneously became an integral part of the classroom learning practices. Students took and shared annotations during the lectures and used them as complementary preparation material for the exam. Second, a correlation was observed between the annotation browsing time and the final exam grade. Third, a social bias emerged in favor of accessing one’s own and friends’ annotations.

Publications

F. Slimane; A. Mazzei; L. Tomasin; F. Kaplan : Text Line Detection and Transcription Alignment: A Case Study on the Statuti del Doge Tiepolo. 2015. Digital humanities, Sydney, Australia, June 29 - July 3, 2015.
Y. Rochat; M. Fournier; A. Mazzei; F. Kaplan : A Network Analysis Approach of the Venetian Incanto System. 2014. Digital Humanities 2014, Lausanne, July 7-12, 2014.
M. M. J.-A. Simeoni : Semi-Automatic Transcription Tool for Ancient Manuscripts ; IC Research Day 2014: Challenges in Big Data, SwissTech Convention Center, Lausanne, Switzerland, June 12, 2014.
A. Mazzei; T. Koll; F. Kaplan; P. Dillenbourg : Attentional Processes in Natural Reading: the Effect of Margin Annotations on Reading Behaviour and Comprehension. 2014. ACM Symposium on Eye Tracking Research and Applications, Safety Harbor, USA, March 26-28, 2014.
A. Mazzei; S. Eivazi; Y. Marko; F. Kaplan; P. Dillenbourg : 3D Model-Based Gaze Estimation in Natural Reading: a Systematic Error Correction Procedure based on Annotated Texts. 2014. ACM Symposium on Eye Tracking Research and Applications, Safety Harbor, USA, March 26-28, 2014.
A. Mazzei / P. Dillenbourg; F. Kaplan (Dir.) : Expanding Eye-Tracking Methods to Explain the Socio-Cognitive Effects of Shared Annotations. Lausanne, EPFL, 2013. DOI : 10.5075/epfl-thesis-5917.
A. Mazzei; J. Blom; L. Gomez; P. Dillenbourg : Shared annotations: the social side of exam preparation. 2013. Eighth European Conference on Technology Enhanced Learning, Paphos, Cyprus, September 17-21, 2013.
Y. Marko; A. Mazzei; F. Kaplan; P. Dillenbourg : A 99 Dollar Head-Mounted Eye Tracker ; In F. Vitu, E. Castet, & L. Goffart (Eds.), Abstracts of the 16th European Conference on Eye Movements. Presented at the ECEM, Marseille., Marseille, August 21-25, 2011.
A. Mazzei; F. Kaplan; P. Dillenbourg : Producing and Reading Annotations on Paper Documents: a geometrical framework for eye-tracking studies ; Symposium N°13: Interacting with electronic and mobile media: Oculomotor and cognitive effects. In F. Vitu, E. Castet, & L. Goffart (Eds.), Abstracts of the 16th European Conference on Eye Movements. Presented at the ECEM, Marseille., Marseille, August 21-25, 2011.
A. Mazzei; F. Kaplan; P. Dillenbourg : Cognitive and social effects of handwritten annotations ; Red-conference, rethinking education in the knowledge society, Monte Verità, Switzerland, March 7-10, 2011.
Q. Bonnard; A. Mazzei; F. Kaplan; P. Dillenbourg : Vocal Sticky Notes: Adding Audio Comments on Paper Documents ; 28th ACM Conference on Human Factors in Computing Systems, Atlanta, Georgia, USA, April 10–15, 2010.
A. Mazzei; F. Kaplan; P. Dillenbourg : Extraction and Classification of Handwritten Annotations ; 1st International Workshop on Paper Computing, a Ubicomp 2010 workshop (PaperComp 2010), Copenhagen, Denmark, September 26, 2010.