A Scalable and Freely Accessible Machine Learning Based Web Application  for the Early Detection of Dyslexia

Summary

Dyslexia is the world’s most common neurological learning disability - it affects 1 in 10 people worldwide. And though it has been proven that an early diagnosis can significantly reduce learning difficulties later in life, screenings for dyslexia remain inaccessible to a majority of the world because of their prohibitive cost ($1000-$2000/screening) and their need for specialized scientific equipment. Previous studies have shown that dyslexics exhibit significantly longer and more frequent fixations while reading than non-dyslexic readers. The goal of this research was to build a free, web-based-application that uses a standard computer webcam to screen a child while reading a passage on the screen. By implementing a novel combination of different machine learning algorithms, this research was able to produce the first-ever, freely available, highly accurate (with a maximum possible error of only a few pixels) eye-tracking methodology for the standard computer webcam. These eye tracking results were then analyzed to determine the duration and frequency of gaze fixations made while reading. Based on this metric, the application was able to predict if a child has a higher risk of dyslexia with an accuracy of 90.18%, as tested on a dataset of real dyslexic patients with 370 samples classified as high or low risk. Because it is completely free and doesn't require any hefty scientific equipment, this application provides the first-ever, freely available, highly accurate test for risk of dyslexia that is accessible to millions of families around the world without regard to financial status or physical location.

Youtube-Summary-Video:https://www.youtube.com/watch?v=f0uRn7npd9A&feature=youtu.be

Question / Proposal

The motivating question for this research was: "Can we build a highly accurate screening application for dyslexia that is available to anyone with a laptop?"

Dyslexia is the world’s most common neurological learning disability - it is estimated that it affects 1 in 10 people worldwide. The term “dyslexia” is used to describe the cluster of symptoms that results in difficulties with comprehension and demonstration of spoken/written language.

Dyslexics encounter tremendous difficulties when learning to read and write alongside their peers, and often eventually completely give up because of the obstacles in their path.

An early diagnosis can be life changing for children with dyslexia: 90% of children with reading difficulties will be able to attain grade level proficiency if they recieve help by the first grade!  However, 75% of students who do not receive help by the age of 9 will continue to struggle with reading throughout their life.

Getting an evaluation and diagnosis of dyslexia, however, is no easy feat. In most cases, families have to pay between $1000 and $2000 for a neuropsychologist screening (often not medically-insured=). Many developing countries don’t have access to these screenings at all.

My goal was to build an application that could detect medically-proven metrics for dyslexia using only the standard-laptop-inbuilt-camera, thereby resulting in an accurate system that can predict whether a child has a high risk for dyslexia, allowing children worldwide to get the help they need and reach their full potential without regard for physical location or financial status.

Research

Other Methods for Screening Dyslexia: Newer methods for the diagnosis of dyslexia have emerged, but they each have serious drawbacks that hinder their scalability. In 2015, Dr. Luz Rello of Carnegie Mellon University used a machine learning model to automatically predict if readers were at a risk for dyslexia. Her model was based on a Support Vector Machine (SVM) binary classifier that trained on a dataset consisting of 1,135 videos (recorded using an eye tracker) of people with and without dyslexia reading. And while this was the first time that a combination of eye tracking and machine learning techniques had been used to identify a risk for dyslexia in practice, the model was relatively inaccurate, with a maximum 80.18% accuracy. It was also not scalable, as it required hefty eye tracking equipment that is only available in scientific laboratories. This makes it inaccessible to a majority of the population across the world.

 

Scientific Basis of Research: Although dyslexia is, in essence, a language based neurological learning disability, numerous studies have shown that eye movements during reading can be highly indicative of one’s dyslexic status. Specifically, readers with dyslexia are known to exhibit both a higher frequency and duration of gaze fixations while reading. (A fixation is the maintaining of the gaze on a single location.) For example, for a given passage, while non-dyslexic readers typically have an average fixation duration of ~250 ms, dyslexic readers have an average fixation duration of more than 350ms. The contrast between the duration of gaze fixations made while reading in a dyslexic versus a non-dyslexic reader is shown in Figure 1 attached (Mynewsdesk, 2017). In the diagram, circles represent a gaze fixation; the larger the radius of the circle, the longer the duration of the gaze fixation.

 

 

Given the drastic difference between the fixation statistics of dyslexics and non-dyslexics, if I build a system that can detect the duration and frequency of fixation in a live webcam stream, I would have built an accurate system to predict whether a child has a high risk for dyslexia. One of the key realizations I had was that this analysis can be performed on eye movement patterns directly- without knowing the exact gaze location on the screen.

Connecting to my Proposed Solution: My proposed solution in this research relies on the analysis of duration and frequency of reader fixations in a webcam stream, which results in an accurate system that can predict whether a child has a high risk for dyslexia.

The result is superior to any previous screening methods because 1) it doesn’t require any hefty scientific eye tracking equipment and can be used from the comfort of any home with access to the internet, 2) it doesn’t require any kind of specialized training to use: any parent who wishes to screen their child can do so without having any scientific knowledge, and most importantly, 3) it is completely free, making it accessible to families around the world without regard to financial status.

Method / Testing and Redesign

  1. This research has 6 main steps: Take a video of a child reading a standard passage displayed on a computer.
    • The paragraph is 10 sentences with simple words that a 7-9-year-old child should be able to read.
  2. Separate video into frames using computer vision functions in OpenCV
  3. Isolating right and left eyes from images
    • I implemented code to perform object detection using Haar feature based cascade classifiers, which is essentially a machine learning based approach which trains a function on many positive and negative images and extracts features such as the ones in Figure 2 below.
  4. Developing the first-ever, highly accurate eye tracker that uses only a computer webcam

    • This is one of the most novel parts of this project! I needed a robust method that didn’t require a massive amount of data and that could accurately track the eye movements with a simple computer webcam. To satisfy these tight constraints, I developed a novel ensemble method of my own and implemented code that uses a combination of the Viola-Jones Algorithm, used to identify the corners of the eyes, and the Timm and Barth’s gradient descent optimization algorithm to detect the center of the pupil. The system detects the corners of the eyes along with the center of the pupil because if the distance that the pupil moves is measured relative to the corner of the eye, head movements and other distractions will not alter the tracking results.

    • A German study showed that the Timm and Barth algorithm, along with the ElSe algorithm, is “robust against various sources of noise, such as illumination or off-axial camera position”, which was crucial when building an eye tracking system designed for use in a home environment where varied sources of noise are almost guaranteed to be present. My eye tracking application is also novel because unlike many other eye-tracking softwares, it does not require any calibration whatsoever. As a final post-processing step, I also implemented noise filtering mechanisms in the code to cleanup noise in the pupil center progression waveform.

  5. Extract fixation frequency and duration features to predict dyslexia

    • My next step was to implement waveform analysis and create a Fixations.vs.Time graph for the recording, as shown below. This relied on a thresholding mechanism, i.e., the pupil center is considered to be fixed until it has moved by more than 2 pixels, which was the error margin for the accuracy of pupil center location.

    • The final step in terms of designing the application was to extract the average fixation duration and frequency features to predict dyslexia. Since fixation duration was used as the dominant feature for the classification model, a “threshold” had to be determined: for an average fixation duration (AFD) less than the threshold, a subject likely would not be dyslexic. For an AFD more than the threshold, a subject likely would be dyslexic.

Results

In order to determine this threshold and test the application, I used a freely available patient dataset from the Karolinska Institute, Sweden (Benfato, et al). The Karolinska Institute is Sweden’s forefront academic and medical institution; the dataset consisted of eye movement recordings from 185 children ages 8 and 9 (since recordings for both the left and right eye were extracted from each child, the dataset contained 370 samples). As described in table below, the dataset contained 194 samples with a high risk for dyslexia and 176 samples with a low risk for dyslexia. 260 samples were used to train the model for determine a dyslexia classification threshold, the remaining 110 were used as the evaluation set to determine the model’s accuracy on this real patient dataset.


In order to determine the threshold, an optimization was done (shown in Figure below) sweeping over a set of possible threshold values, from the AFD value of Low Risk children (325 milliseconds) to the AFD value of High Risk children (525 milliseconds), as the threshold for classifying between the two must be in between those two numbers. After analyzing the accuracy percentages for high risk diagnosis and low risk diagnosis across
different thresholds (accuracy rates were determined from the 260 training samples from the dataset), a threshold of 385 milliseconds was determined to be optimal in terms of the tradeoff of specificity vs sensitivity as shown in Figure  below. Therefore, if a subject exhibits an average fixation duration of less than 385 milliseconds, they are characterized as not dyslexic. If a subject exhibits an average fixation duration of more than 385 milliseconds, they are characterized as dyslexic.

I tested this model on the 110 testing samples from the dataset: with a threshold of 385 milliseconds, the model achieves a sensitivity of 90.18% (accuracy for correctly classifying subjects with dyslexia as dyslexic). These steps are all encapsulated in a simple script which is called within an application and can also be invoked with a simple command line interface (CLI) for an end to end dyslexia application.

Results:
- The eye tracking application developed and implemented in this research is highly accurate and has a maximum error rate of only 2 pixels.
- The final application has a sensitivity (accuracy of classifying subjects with dyslexia) of 90.18%, when tested on real dyslexic patient dataset .
- The final application has a specificity (accuracy of classifying subjects without dyslexia) of 85.03%

Conclusion

Conclusions and Future work:

This research provides the first-ever early detection system for dyslexia that is highly accurate, completely free, and can be conducted from the comfort of one’s home! 

The result is superior to any previous screening methods because:

  1. It does not require any hefty scientific eye tracking equipment, and can be used from comfort of any home around the world. 
  2. Does not require any specialized training, and so any parent who wants to screen their child can do so.
  3. It is completely free, which makes it available to families around the world without regard to financial status.

Testing with real patient data as well as tremendous support from the special education and medical communities has confirmed that this application has a real possibility of being a game changer for millions of families with dyslexic children around America and the world, thus empowering them to reach their full potential so that a treatable condition such as dyslexia does not hold them back from pursuing their dreams.

A future alternative method that I am experimenting with to increase accuracy beyond 90% is to convert the pupil movement plots of the eye movement recordings to images using recurrence plots, as shown below. 

Then, I will train a Neural Network model to classify recurrence plot images as either high risk or low risk for dyslexia.

 

Additionally, the eye tracking methodology innovated in this research has tremendous potential to democratize access to screenings for many other neurological conditions such as autism, alzheimers, and more! All of these conditions have been proven to have irregular eye movement patterns, and given that this eye tracking methodology requires nothing more than a simple inbuilt computer webcam, it can be used for screening applications for widespread conditions as well!”

 

About me

Hi! I'm Isha Puri! 

I have loved mathematics all my life: as a young kid, I was obsessed with logic puzzles, and my creative upbringing with STEM (with experiences in math competitions and initial introductions to innovation through household tinkering and First-Lego-League-Robotics) hooked me on the thrill of innovation, creativity, and discovery - I have loved STEM ever since!

In 2016, I had the opportunity to attend the second-ever Stanford AI Lab Summer Program, where I was immersed in Artificial Intelligence and its incredible creative and societal applications. Ever since then, I have taught myself as much as possible about computer science and AI via online tutorials and independent research projects (such as an Natural-Language-Processing-based tweet classifier to assist people during natural disasters, and an application that alerts power companies to down wires), and I hope to continue to study computer science and innovate cool applications that benefit society in college!

I am most proud of my web-based application for early detection of dyslexia that I presented here-  it is a project I developed and refined over this year. Winning prizes at the Google Science Fair would mean the world to me: I am so passionate about creating applications that have the potential to benefit society, and I am so excited at the opportunity to immerse myself in the incredible scientific journeys and experiences that the GSF offers.

Please check out my Maker Portfolio!

 

Please visit my website to see a full compilation of my experiences/research-projects/honors: ishapuri.tech!

Health & Safety

No wet lab work was conducted, and so health/safety procedures were not required.

My mentor Dr. David Cox (david.cox@seas.harvard.edu) Professor, Harvard University - Brain Science Institute introduced me to the exciting research field of eye tracking algorithms and their applications to neuroscience. I was fascinated by this field and after my internship, I took the initiative to research the applications of eye tracking further. I was excited to find the connection between abnormal eye movements to dyslexic children, and took it upon myself to develop an eye tracking algorithm for a standard computer webcam to detect dyslexia!

The dataset of 370 eye recordings from dyslexic and non-dyslexic children used was a publicly available, de-identified dataset published online by the Karolinska Institute in Sweden. 

Bibliography, references, and acknowledgements

Acknowledgments:

I would like to thank my mentor Dr. David Cox of Harvard University’s Brain Science Institute. He introduced me to the fascinating world of eye movement analysis via computation when I worked in his neuroscience lab during the summer of 2017. While there, I built an application that tracked the eye movements of rodents and analyzed their implications on the rodent learning patterns.

I would also like to thank:

  • MIT THINK, for their generous guidance and for giving me the opportunity to present my research in front of MIT professors and graduate students.
  • AI4ALL, for providing me funding and support to present my research at the 40th Annual IEEE Engineering in Medicine and Biology Conference and get important feedback to refine my research further.
  • Dr. Heidi McCarthy, the Director of Special Education in the Chappaqua Central School District, for her tremendous support of this research.
  • Trice and Jimmy Hallac of Chappaqua, NY for allowing me to hear their family’s stories about their journey with a dyslexic child

 

References:

Alexander-Passe, N. (2006). How dyslexic teenagers cope: an investigation of self-esteem, coping and depression. Dyslexia, 12(4), 256-275. doi:10.1002/dys.318

Benfatto, M. N., Seimyr, G. Ö, Ygge, J., Pansell, T., Rydberg, A., & Jacobson, C. (2016). Screening for Dyslexia Using Eye Tracking during Reading. Plos One, 11(12). doi:10.1371/journal.pone.0165508

Eye movements during reading [.mov]. (2017, November 2). Mynewsdesk.

Francis, D. J., Shaywitz, S. E., Stuebing, K. K., Shaywitz, B. A., & Fletcher, J. M. (1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88(1), 3-17. doi:10.1037//0022-0663.88.1.3

Frequently Asked Questions. (n.d.). Retrieved January 01, 2018, from https://dyslexiaida.org/frequently-asked-questions-2/

Fuhl, W., & Santini, T. (2017). PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection. PupilNet v2.0: Convolutional Neural Networks for CPU based real time Robust Pupil Detection. doi:10.1117/12.2254647.5361821944001

Fuhl, W., Geisler, D., Santini, T., Rosenstiel, W., & Kasneci, E. (2016). Evaluation of state-of-the-art pupil detection algorithms on remote eye images. Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct - UbiComp 16. doi:10.1145/2968219.2968340

How much does a learning disability evaluation cost? (2017, May 25). Retrieved January 01, 2018, from http://drmessler.com/learning-disability/much-learning-disability-evaluation-cost/

Hyönä, J., & Olson, R. K. (1995). Eye fixation patterns among dyslexic and normal readers: Effects of word length and word frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition,21(6), 1430-1440. doi:10.1037//0278-7393.21.6.1430

Puri, I., Cox, D. (2018). A System for Accurate Tracking and Video Recordings of Rodent Eye Movements using Convolutional Neural Networks for Biomedical Image Segmentation. Proceedings of 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 40, 3590-3593.

Rello, L., & Ballesteros, M. (2015). Detecting readers with dyslexia using machine learning with eye tracking measures. Proceedings of the 12th Web for All Conference on - W4A 15. doi:10.1145/2745555.2746644

Riddick, B. (2016). Living with dyslexia: the social and emotional consequences of specific learning difficulties/disabilities. New York: Routledge.

Tai, Y. (n.d.). Dyslexia and Eye Movements. Retrieved November 1, 2018, from http://commons.pacificu.edu/cgi/viewcontent.cgi?article=1019&context=vpir

Timm, F., & Barth, E. (2011). Accurate Eye Centre Localisation By Means Of Gradients. Proceedings of the International Conference on Computer Vision Theory and Applications. doi:10.5220/0003326101250130

Vellutino, F. R., Scanlon, D. M., Sipay, E. R., Small, S. G., & Al, E. (1996). Cognitive profiles of difficult-to-remediate and readily remediated poor readers: Early intervention as a vehicle for distinguishing between cognitive and experiential deficits as basic causes of specific reading disability. Journal of Educational Psychology, 88(4), 601-638. doi:10.1037//0022-0663.88.4.601

Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. doi:10.1109/cvpr.2001.990517

Widell, Victor. "Dsxyliea." Geon.github.io. N. p., 2016. Web. 1 Nov. 2018.