POHA: Personal Oral Health Advisor with a multimodal sensor, mobile cloud application, and machine learning

Summary

The mouth is considered a mirror of the overall health of the body and yet there is no personal device available today that can be used at home to monitor oral health. The objective of this project is to create a cost-effective, multimodal, personal oral sensing device that automatically classifies sensed data and provides interactive advice about oral health. After surveying dentists about the kinds of information that would be useful, I created POHA, a handheld home device that integrates image, video, smell, temperature, and 3D orientation sensing, and communicates with a smartphone via USB and Bluetooth. POHA scans the mouth and teeth of a person while they perform a brushing-like motion. I developed image and smell classification models by training machine learning algorithms with hundreds of images collected from the web and air quality values of various mouth odors obtained with a low-cost gas sensor. My first experiments show 85% accuracy in detecting dental cavities, 77% in detecting gingivitis, and 85% in detecting bad breath. I developed a mobile application that provides users real-time views of sensed data, shows classification results and advice, collects health information, and stores tagged sensor data in a central cloud database. This first-of-its-kind device fits easily into one's daily  routine, provides preventive advice, and encourages seeing a dentist. POHA also provides doctors for the first time with tagged daily oral sensing data from millions of people that would be invaluable for medical research in detecting, preventing, and predicting numerous diseases that have oral manifestations.

Question / Proposal

I am trying to find out if it is possible to create an affordable and portable personal device that can be used by people at home to frequently monitor oral health, automatically detect oral problems, and provide relevant advice, while also capturing valuable data that is not available today to doctors and medical researchers. Today there are different kinds of clinical oral scanners, and home devices for monitoring other aspects of health, but I did not come across a personal oral health advisor. I expect that it is very much possible to create such a device taking advantage of low-cost electronic sensors, microcontrollers, smartphones, and 3D design technologies available today. I also want to investigate how machine-learning algorithms could be trained to detect oral health problems – starting with dental cavities, gingivitis (inflamed gums), and halitosis (bad breath). I want to store sensed data in a cloud database that scales to millions of users that would potentially use the device, so that doctors and researchers can get insights into the evolution of their patients’ health, provide remote advice, and potentially come up with new models for detection of other health problems. My experiments will first investigate relevant electronic components and integration techniques to build a viable portable device. Next I will explore if machine-learning algorithms can indeed be trained to detect oral problems. I am hoping to achieve at least 75% accuracy in detection. I would also like to get validation from dentists and doctors that my device is useful.

Research

This project started when I got the idea that it would be useful for me and my family members to know if we were brushing our teeth properly and taking proper care of them without getting cavities and other dental problems. To pursue this idea further, I went to a well-known dentist who lives in my community. I prepared a survey and asked her many questions about personal dentist tools and discovered that there was no personal tool that allowed someone to do an oral checkup on their own. This prompted me to develop such a device. 
I looked into various websites and research articles that show the seriousness of oral health problems in the world today [2,3,6, 8,9]. These showed that over 3.9 billion people suffer from oral health problems [6] and that 60% of children in the US have a cavity by the age of five [8]. In some countries 90% of school children have oral cavities [9]. I also found many articles [1,4,5,7] that talked about the strong connection between oral health and the overall health of the body. These indicated to me that monitoring oral health was important not only for dental problems but also for many other health conditions.  I also looked into the state of oral health care in developing countries like India [11,12] and was surprised to find that there is only one dentist for every 150,000 people in rural India, which has 70% of India’s population. This research further motivated me to pursue POHA. 
I looked extensively into oral scanners that are available today [13,14,15,16,17].  I found that there were several scanners used in dental clinics to get 3D images of the teeth, plan dentures etc., but that there was no personal oral scanner that could be used at home. The intelligent brush in [17] provides a user advice on how to better brush their teeth but still does not provide a view of the condition of their mouth. Even clinical scanners were not performing automatic classification of images and were not capturing breath. I did extensive research on odor and air quality detection [23,24,25, 26,27,28,29,30,31,32], standards for olfaction [33,34,35], and gas sensors [36,37,38,39,40,41,42,43]. I also researched cameras and other electronic sensors [46,47,49,50] and integration techniques [44,45,48] . 
I studied research papers on convolutional neural networks [18, 20] for image classification and machine learning techniques and tools [19,21] and decision classification trees [22].  As I got into developing a mobile app and cloud platform I learnt Android development tools [52,53,54] and the Google Cloud Platform and real-time database [55,56,57].  I also learnt about 3D modeling tools [52] to build and print a 3D enclosure for POHA. 
I also learned a lot from my mentors who spanned the fields of dentistry, electronics engineering, mechanical engineering, and computer science. 
As a result of all this research and in the course of the project, I came up with an overall solution design for POHA as illustrated in Figure 1.

 Fig. 1

Method / Testing and Redesign

I built a simple version of POHA with a waterproof USB endoscope camera of 20 mm focal length with adjustable LED lighting, and integrating it into an electric toothbrush by detaching the brush head and replacing it with the camera and encasing it with the plastic case of the brush head ($10 cost)(Figure 2).

Fig. 2

I then built an advanced version ($45 cost) that includes a smaller camera, gas sensor, temperature sensor, three accelerometers, an Arduino Nano microcontroller, a Bluetooth chip, and a rechargeable 3.8V battery. I integrated these components into a PCB (Figure 3).

Fig. 3
I designed a 3D enclosure (Figure 4) for convenient use at home with a tube to hold the camera that can easily be placed into the mouth for scanning while the accelerometers would measure orientation. The response of the gas sensor is good when the user blows directly into it.

Fig. 4

I positioned the gas sensor on the side of the brush-like device for easy blowing.The temperature sensor is placed close to the gas sensor to measure the ambient temperature, as the gas sensor is sensitive to temperature. I designed a holder for the temperature sensor to be next to the gas sensor while being removable to measure body temperature.
I tested the device (Figure 5) to ensure all sensors and the USB and Bluetooth communications were working. I left the device switched on for three days to ensure it did not break down.

Fig. 5
For automatic detection of cavities, gingivitis, and bad breath from POHA sensor data, I selected a convolutional neural network based image classifier (from AutoML) and a decision tree classifier (from BigML) for smell classification. I used 100 images each of healthy teeth and gums, teeth with cavities, and gums with gingivitis to train the image classifier (Figure 6) with public images found on the web.

Fig. 6

I used four values from the gas sensor to train a decision tree for distinguishing good and bad breath. These were the conductance of the sensor, the ratio of the current conductance to its initial conductance, rate of change of conductance over 100 ms, and the conductance in a reference fresh air condition. Since there is no public data available for breath, I collected samples of my own breath and that of 3 family members (with permission) using the POHA device five times a day for 20 days and saved the readings. For each breath sample I had three people classify the breath as good breath or bad breath and used this labeled data to train the decision tree classifier. I separated training and test sets for images and breath samples.


I wrote a mobile app on an Arduino smart phone to read sensor data from the USB, display live images to the user, capture additional data from the user, store data in a Firebase real-time database on Google Cloud, read image and breath classification results from the cloud, and display results and advice to the user.

Results

1. My first significant result is that I succeeded in creating two working prototypes of the POHA device. The first costs just $10 and involves only a camera and a modified toothbrush while the second costs $45 and senses video, 3D orientation, smell/air quality, and temperature. This prototype is not as compact as it could be because I wanted flexibility in accessing the components and modifying the programs on the microcontroller but is still in a handheld portable form that consists of a rectangular box of dimensions 18 cm (l) x 3.5 cm ( w) x 4 cm (h) with an additional protruding narrow tube of diameter 1.2 cm and length 5 cm. The device runs on a 3.8 V battery with 200 mAh and weighs 90 grams. 
2. I first tested the classification of dental cavities and gingivitis with images obtained from the POHA device using scans of my own mouth and those of 3 family members. I found that POHA consistently correctly classified the teeth of my family members as not having cavities or gingivitis in 200 trials and also correctly detected the cavity that one family member had in 100 trials. This also showed that the classification trained with images from the web could still be applied to images from the POHA device. To further test the classification, I tested the automatic detection of dental cavities and gingivitis using images I obtained from the web, which were distinct from those I used for training. I intentionally chose images that needed my close examination to classify correctly as I considered this a more challenging test for the system and also indicates if the system can detect problems early when a human is just able to detect these.  Tables 1, 2 below show the classification results for dental cavities and gingivitis on these image sets.  Figures 7, 8 show examples of images of gums and cavities being classified correctly. 

  Table 1, 2

 


Fig. 7, 8


Figure 9 shows the decision tree classifier for breath classification obtained from the training data based on four parameters described in the earlier section.  Using this decision tree I was able to classify good and bad breath and compare the results to human subjective classification of breath. Table 3 shows the results of classification of breath based on these experiments.  Figure 10 shows examples of good breath and bad breath samples (as classified by a human) that were used in these experiments.

Table 3

Fig. 9

Fig. 10

3. In 500 trial scans that I performed, each lasting two to three minutes, the Mobile app successfully read POHA sensor data at 20 frames a second and stored all sensor data in the real-time Cloud database after the scan. Classification results were obtained within 90 seconds after the scan.
I also demonstrated POHA to four dentists and two doctors. All six agreed that they are unaware of another home device like this and that patients’ home monitoring data from such a device would be invaluable.

 

Conclusion

The results were promising and showed that we could indeed build a Personal Oral Health Advisor using multimodal sensors that can be used by people at homes. The POHA device is a first-of-its-kind home device and is affordable at a cost of $45. Even the simpler version of the device that costs only $10 has significant value as it provides the core functions of scanning the user’s mouth, automatically detecting cavities and gingivitis, and advising the user, while storing all sensed data in a real-time cloud database. 
The automatic classification results were encouraging and are a good starting point for providing useful advice to the user. As the false negatives were significantly lower, the results indicate that a case where a user has a cavity or gingivitis or bad breath is unlikely to be missed while the relatively higher false positives indicate that there would be cases where a user is warned of a potential problem even when they did not have one. I think the accuracy of the system would be improved by significantly increasing the volume of training data.   I was constrained by the limited public data available on the web, even for images of cavities and gingivitis, while there was no oral breath data available at all. I hope to obtain access to larger data sets of images available in the dentist community and also gather more training data in a dental clinic with the permission of patients. The breath data could be used to create new data sets that do not exist today. 
I think one of the most powerful aspects of the POHA system is the creation of new data sets.  Dentists and doctors that I talked to were quite excited about this potential as they can start looking at how the oral condition of a patient evolves during the onset of a disease. Doctors do not have such data sets today. Doctors also advised me to train the system to recognize other kinds of problems beyond cavities, gingivitis, and bad breath.  Works such as  [4,7] indicate that there are very clear and distinct oral manifestations  (on the lips, tongue, gums, teeth) of various health problems including gastrointestinal disorders, blood diseases, endocrinal disorders, nutritional deficiencies, viral, fungal, and bacterial infections. By providing a way to capture frequent oral data from people, POHA opens up the potential to detect and provide early warnings for many of these diseases. 
Dentists were also positive about the use of breath in my experiments. They said that they are able to detect diseases such as diabetes merely by smelling a patient’s breath. This encourages me to further research and pursue breath based disease detection. 
Some dentists and doctors asked me to look into the sterilization and sanitization of the POHA device. While I took basic precautions such as wiping the device and providing a waterproof camera that can be washed, I am encouraged by this feedback, which shows interest in using the device and making it more safe and practical. 

About me

My name is Lalitha Pingali. I am 14 years old and live in Bengaluru, India. I really like to help people with problems that are affecting their everyday lives - garbage, stink, teeth problems, and illiteracy in rural villages are some of them.  I got into STEM when my older sister started doing important research on the heavily polluted lakes in Bengaluru. This inspired me to find other problems on my own that I could solve using science. One scientist who inspires me is Aryabhatta. Back when there was not even a proper numerical system, Aryabhatta came up with the idea of a number called 'zero'. He questioned simple shapes like triangles, which gave birth to subjects like trigonometry, and he created the place value system. He showed that through questioning and observation we could come up with solutions. 
Last year I did a project that creates a Stink Map of any environment. This project won a top award in the IRIS National Science Fair and I was selected as a Broadcom International delegates from India to go to the International Science and Engineering Fair in Pittsburgh where I was thrilled to meet brilliant kids from all over the world. This experience motivated me to apply to the Google Science Fair. I truly think the POHA project is a game changer as it could touch almost everyone in the world. Winning at the Google Science Fair would give this project that recognition and impetus to make a real difference. 

Health & Safety

I did not have access to people with teeth problems, so I used images that were publicly available on the web. For mouth odor testing, I only tested on myself and a few willing family members with their consent as there was no publicly available data on mouth odor on the web. 

Safety precautions I followed:

-Not touching the camera to living tissues in the mouth

-Frequent wiping of the camera, temperature sensor and smell sensor after every scan with sterile wipes to keep it clean

-Covering the temperature sensor with x-ray sleeves to avoid contamination

My device is securely enclosed, and the camera and thermometer are waterproof, so these parts that are used to scan the mouth can be cleaned with wet wipes and even washed

 

My mentors are:

Dr.Sangeetha Honnur:

Contact details are: 

    -Smile Elements Dental Clinic, Bellandur, Bengaluru

    -E-mail:sangeeta_viru@yahoo.com

Mr.Abulais:

Contact details are:

    -Sai Vigyan Institute, Bengaluru

    -E-mail:abulais3@gmail.com

Mr. Arpan Gadia:

Contact details are:

    -Rocket Pathshala, Bengaluru

    -E-mail:hello@rocketpathshala.com

 

Bibliography, references, and acknowledgements

I would like to thank Dr. Sangeetha Honnur and her team of dentists at Smile Elements Dental Clinic, Mr.N Arumugam and Mr.Abu Lais from Sai Vigyan for teaching me Arduino and electronics, Mr.Anand from Intellitrak for teaching me Android and Java programming and Mr. Arpan Gadia from Rocket Pathshala for teaching me how to use Fusion360 software for 3D printing. I would also like to thank my parents and grandparents for their constant encouragement and for reading my project report and providing lots of feedback.

REFERENCES

ORAL HEALTH

1.     https://www.webmd.com/oral-health/features/oral-overall-health

2.     http://www.homd.org/

3.     https://www.ncbi.nlm.nih.gov/pubmed/11314311   

4. https://www.researchgate.net/publication/233391502_Mouth_as_a_mirror_of_systemic_diseases

5.     www.eadph.org/.../Mouth%20The%20Mirror%20of%20the%20Body_Camilleri.pdf

6.     www.worldoralhealthday.org/sites/default/files/assets/2018_WOHD-brochure-EN.pdf

7.     https://www.slideshare.net/neetulove16/oral-cavity-is-the-mirror-image-of-body

8.     http://www.aapd.org/assets/1/7/state_of_little_teeth_final.pdf

9.     https://www.dentistryiq.com/articles/2014/03/almost-100-of-adults-and-60-to-90-of-schoolchildren-worldwide-suffer-from-dental-caries.html

10.  https://www.nidcr.nih.gov/research/data-statistics/dental-caries/children

11.  Oswal K, Gour R, Deshpande T, Dhoble A (2017) Developing an Innovative Model of Oral Health Service Delivery in India. J Prev Med Healthc 1(4): 1014

12.  Om PrakashKharbanda&KunaalDhingra   Oral health inequality in India: Perspectives and solutions, natureIndia, 20 March 2017

 

ORAL SCANNERS AND BRUSHES

13.  Dentalwings intraoral scanner(www.dentalwings.com/products/intraoral-scanner/)

14.  VIZ intraoral scanner(https://www.adin-implants.com/product/intra-oral-scanner/)

15.  TRIOS Intraoral Scanner(https://www.3shape.com/en/products/trios/intraoral-scanners)

16.  Aadva Intraoral Scanner(https://www.gceurope.com/products/aadvaios/)

17.  https://shop.colgate.com/?ref=colgate.com

 

MACHINE LEARNING

18.  YannLeCun, YoshuaBengio& Geoffrey Hinton   Deep Learning, Nature volume 521, pages 436–444 (28 May 2015)

19.  https://cloud.google.com/vision/automl/docs/beginners-guide

20.  https://console.bluemix.net/docs/services/visual-recognition/index.html#about

21.  https://bigml.com/

22.  https://towardsdatascience.com/a-beginners-guide-to-decision-tree-classification-6d3209353ea

 

ODOR DETECTION

23.  E. A. Fox & V. E. GexProcedure for Measuring Odor Concentration in Air and Gases, Journal of the Air Pollution Control Association, 7:1, 60-61, DOI:10.1080/00966665.1957.10467783]http://www.tandfonline.com/doi/abs/10.1080/00966665.1957.104677834

24.  Amos Turk, James W.Johnston, David G.Moulton, Human Responses To Environmental Odors, Academic Press New York and London(1974)

25.  Chemela C, Riesenmey C, Batton-Hubert M, Vaillant H. Odour-impact assessment around a landfill site from weather-type classification complaint inventory and numerical simulation. Journal of Environmental Management 2012;93:85–94.

26.  Hadrich J, Wolf C. Citizen complaints and environmental regulation of Michigan livestock operations. Journal of Animal Science 2011;89:277-86.

27.  Munoz R, Sivret E, Parcsi G, Lebrero R, Wang X, Suffet IH, Stuetz R. Monitoring techniques for odour abatement assessment. Water Research 2010;44:5129–49.

28.  Schiffman SS, Williams CM. Science of odors as a potential health issue. Journal of Environmental Quality. 2005; 34:129–38.

29.  Steinheider B. Environmental odours and somatic complaints. International Journal of Hygiene & Environmental Medicine. 1999 Aug; 202:101–19.

30.  Schiffman SS, Gutierrez-Osuna R, Nagle TH. Measuring odor intensity with e-noses and other sensor types. Proceedings of the 9th International Symposium on Olfaction and Electronic Nose; 2002 Sep 29–Oct 2; Rome, Italy.

31.  Guidelines On Odour Pollution And Its Control, CENTRAL POLLUTION CONTROL BOARD Ministry of Environment & Forests, Govt. of India(May 2008)http://cpcb.nic.in/upload/NewItems/NewItem_141_package_odourreport_2.12.08.pdf

32.  Charles W. Gruber , George A. Jutze& Norman A. Huey Odor Determination Techniques for Air Pollution Control, Journal of the Air Pollution Control Association, 10:4, 327-330, DOI: 10.1080/00022470.1960.10467940 http://www.tandfonline.com/doi/pdf/10.1080/00022470.1960.10467940

33.  EN 13725 European Standard for Olfactometry

34.  International technical and air quality standards prEN 264086, VDI 4257 Part 2, ATTA TS1 and EN 13829

35.  https://www.olfasense.com/en/Environmental-odour-management?gclid=EAIaIQobChMI--KNs5Gc1QIVRYePCh3JRwTAEAAYASAAEgJ25PD_BwE#fndtn-tabfieldadditionaloffers11

36.  Figaro sensors’ 2620.http://www.figarosensor.com/products/

37.  FigaroTGS2444 for ammonia detection (10 to M300 ppm)

38.  Figaro TGS2602Air contaminants/odor monitoring (VOCs, ammonia, H2S, etc.) (1 - 30ppm of EtOH)http://www.amazon.in/SainSmart-TGS2602-Contaminants-Detector-Raspberry/dp/B00KCEQTXG

39.  SainSmart MQ135 DC 5V 10-1000ppm Ammonia Gas Sensor Module Detector for Arduinohttp://www.amazon.in/SainSmart-10-1000ppm-Ammonia-Detector-Arduino/dp/B00E5Y57SW/ref=pd_sim_147_5?_encoding=UTF8&psc=1&refRID=5HKV5KTZ4JYBMNXEBAQW

40.  http://pdf.directindustry.com/pdf/synkera-technologies/mikrokera-4l-hydrogen-sulfide-sensor-p-n-727/54300-609117.html#open.

 

41.  https://community.particle.io/t/voc-indoor-air-quality-sensor-iaq-as-mlv-from-appliedsensor/11441/3

42.  https://www.olfasense.com/en/Environmental-odour-management?gclid=EAIaIQobChMI--KNs5Gc1QIVRYePCh3JRwTAEAAYASAAEgJ25PD_BwE#fndtn-tabfieldadditionaloffers11

43.  Gonzalez-Jimenez J, Monroy JG, Blanco JL. The Multi-Chamber Electronic Nose--an improved olfaction sensor for mobile robotics. Sensors (Basel). 2011;11(6):6145-64

 

 

ELECTRONICS, SENSING, AND 3D MODELING

44.  https://store.arduino.cc/usa/arduino-nano

45.  https://components101.com/microcontrollers/arduino-nano

46.  https://www.amazon.com/DEPSTECH-Endoscope-Waterproof-Inspection-Adjustable/dp/B0749BQG1B/ref=sr_1_14?ie=UTF8&qid=1543753813&sr=8-14&keywords=endoscope+borescope

47.  https://rees52.com/arduino-sensors/95-gy-521-mpu6050-module-3-accelerometer-for-arduino-ar009

48.  https://howtomechatronics.com/tutorials/arduino/arduino-and-hc-05-bluetooth-module-tutorial/

49.  http://www.figarosensor.com/products/2602pdf.pdf

50.  https://www.maximintegrated.com/en/products/sensors/DS18B20.html

51.  https://www.autodesk.com/products/fusion-360/overview

 

MOBILE APPS AND CLOUD PLATFORM

52.  http://appinventor.mit.edu/explore/library.html

53.  https://developer.android.com/docs/

54.  https://docs.thunkable.com/

55.  https://cloud.google.com/docs/

56.  https://cloud.google.com/storage/

57.  https://firebase.google.com/docs/database/