Smart Necklace

Resumo

This study presents Smart Necklace, a non-intrusive intelligent detecting device which measures any movement created by the muscle in our neck, such as drinking, eating or talking. It can not only detect whether we cough or sneeze but also measure the wearer's pulse. This technology can be used by anyone, including those with mental or physical health problems. It is worn like a jewelry without causing any discomfort, with all possible fashionable design waiting for it.

The method is based on a piezoelectric sensor, which requires no external power source or recharging. Without relying on any ultrasound or infrared waves, the sensor can track the vibrations created by the movements of the muscles in our neck when we eat or drink or talk.

With different movements causing different pulses of electricity, the piezoelectric film sensor can turn these vibrations into tiny electrical pulses. The device can even train itself to improve accuracy with deep learning model CNN (Convolution Neural Network), which allows Smart Necklace to identify what action is taking place.

Smart Necklace can make contribution to an aging society where a huge portion of the elderly have to live alone. In the future, it can be connected with a smart phone, and send out information or warnings in real time to help caretakers know whether the elderly is drinking enough water or eating too much food or having life-threatening problems.

Pergunta / proposta

Questions:

1. How does this Smart Necklace help human health?

2. How to make a device that can collect the data needed but isn't harmful to our body?

3. How can Smart Necklace classify whether the tester is drinking or eating or talking?

4. How can people be motivated to wear this device all day?

 

Hypotheses:

There are numerous studies on the body about the benefits of drinking water, as well as consuming too much food or not eating at regular intervals. Studies also indicate that talking too little may also cause mental health problems. Smart Necklace is a device that can monitor your health based on the data it has collected. Since this device needs to be worn all day, it cannot be dangerous to the human body. So in order to promote safety, Smart Necklace doesn't have any sensors that transmit ultrasound or infrared waves. Using a piezoelectric sensor meets our requirements, and when we eat or drink or talk, these movements require the muscles in our neck to move. The Piezoelectric film sensor turns these vibrations into tiny electrical pulses, and the data can in turn be used for machine learning. This data can also be used by signal processor such as a spectral analyzer to extract useful information directly from the data itself. The analysis results can help humans to understand whether they are at or nearing a potential health risk. While some people may be concerned that the device is too big or awkward to wear, it is important to choose a film sensor to that is fashionable to the wearer.

Overall, the study is focused on making a safe, fashionable, wearable, intelligent detecting device – Smart Necklace.

Pesquisa

Factors which impact human health status:

Eating disorders [1][2] : Eating disorders, such as anorexia nervosa or overeating, not only cause your weight to dangerously fluctuate, but may also be a factor in mental health problems. The study shows that both stress and eating are related to increased anxiety and depression.

Drinking water [3][4] : There are a whole host of benefits to drinking water sufficient water; such as the prevention of headaches, kidney stones as well as increasing physical performance.

Heart rhythm problems [5] : There are many types of heart arrhythmias, such as bradycardia and tachycardia. These diseases may cause conduction blocks, along with shortness of breath, excessive sweating, and a fluttering in your chest.

Talking too little [6] : Talking too little may be a sign of autism. The study shows that people that tend to talk in single words or don’t like to interact with others are symptoms of autism.

Chronic cough [7] : Coughing may sometimes be uncomfortable, but it is actually a necessary behavior. Most coughs are short-lived, which will see recovery after a few days. If coughing persists for weeks or months, serious problems may be happening to your system. The causes of chronic cough include lung cancer, bronchiectasis, asthma and blood pressure drugs.

These movements all cause tiny vibrations on a human’s neck area, each with a different vibration scenario. These can be measured by high sensitive piezoelectricity materials.

 

Piezoelectricity

Piezoelectric Effect [8]  is the ability to generate an electric charge to mechanical stress. A power source is not needed for this effect to take place. In the other words, you can transfer the vibration into electricity. There are two formulas explaining this phenomenon.

Figure 1 piezoelectric sensor

 

Formula 1

Formula 2

When we eat or drink or talk, these movements all require the muscles of our neck to move, and every movement will create a vibration. According to the formula, the vibration can be transformed into different levels of electricity. This can help us classify the data.

Figure 2 The collected data (talking)

Figure 3 The collected data (drinking)

 

Signal Process

1.  Sampling [9] : The output signal of piezoelectric is an analog signal. This signal needs to be converted into a digital signal for use in future analysis. The equation expresses the conversion process.

Formula 3

 is the sampling period, it determines the resolution of the signal. The sampling frequency fs needs to be greater than two times the bandwidth of the target analog signal in order to meet Nyquist’s Theory.

 

(a) Analog signal x(t) sampling period                 (b) sampling rates sampling period

Figure 4 Sampling result of analog Signal under sampling period

2.  Window Function [12] : the function helps the signal process to mitigate a signal if there are any non-coherence impacts when executing the Fourier Transform. The Hanning window function will be used in the study.

Figure 5 Hanning window function

3.  Spectral Analysis : this provides a different perspective view for a captured signal. It uses FFT (Fast Fourier Transform) [9] to convert the captured signal time-domain to frequency domain.

Formula 4

 

Artificial Intelligence

Machine Learning [10] : is a category of algorithms that allows software applications to acurrately predict outcomes without being programmed. The basic premise of machine learning is to build algorithms which have the capacity to receive input data and can use statistical analysis to predict an output.

CNN [11] : is one of the main categories to do images recognition, images classifications. Convolution Process is a kind of feature extraction for input images. In this study, the input signal is one dimensional data so 1D convolutional layers were used to construct the model.

Figure 6 CNN model

Método / Testes

System Development

1.  Creation of Smart Necklace test environment : It is easy to capture tiny vibration signals from piezoelectric sensors and show the captured waveform on an oscilloscope. The sampling period is set to 2e-4 second and the number of capture points is 36,000, which is 7.2 seconds in duration. The duration can meet the minimum requirements for humans to make each movement.

Figure 7 Smart Necklace testing environment

Figure 8 Block diagram of Smart Necklace testing environment

2.  Block diagram of smart necklace integration version: The integrated version replaces the scope using an MCU-based controller. The controller consists of an op-amp for level shifting, an external SD card for data storage and a Bluetooth module for communication in between Smart Necklace and a smart phone app for future application.

Figure 9 Block diagram of smart necklace integration version

 

Machine Learning

1.  CNN model design: one dimension of convolution will be used in this study consisting of two 1D-convolution layers followed by a maximum pooling layer. Then, full connection classifier layers consisting of four neural network layers will be used for classification.

Figure 10 Flowchart of proposed CNN model in this study

Figure 11 Proposed CNN model in this study

2.  Dataset process: In order to not lose any information at the signal boundary, the method using offset raw data to generate new a dataset will be used in this study. As well, the time domain signal data, frequency domain dataset are also involved in the dataset generation process.

Figure 12 Dataset process

 

Data Collecting

This study focused on collecting the data for training. Through machine learning, our device can classify data accurately. In this experiment, the data is classified into these five different categories.

1.     Heart rate (Static)

2.     Consumption of water

3.     Consumption of food

4.     Oral speech

5.     Coughing

In this experiment, I used The Smart Necklace to collect data. There are 500 train datasets for each category (Except coughing, this is do to the physical discomfort when asked to cough). Figure 13 is photo during the experiment, Figures 14 to 18 are figures of five sets of collected data, which is also from five different categories.

Figure 13 Picture when collecting data

Figure 14 Data of heart rate

Figure 15 Data of consumption of water

Figure 16 Data of consumption of food

Figure 17 Data of oral speech

Figure 18 Data of coughing

 

The appearance of Smart Necklace

Since the user will wear Smart Necklace all day, the device should be safe and pose absolutely no health risk, along with having a nice appearance. The first version of Smart Necklace is designed to look like a fashion choker, this style of choker is very popular this year. Figure 19 and 20 are the final design.

Figure 19 Smart Necklace (front)

Figure 20 Smart Necklace (back)

Resultados

Heart Rate Measurement

The collecting data for heart rate dataset can be used to measure the speed of heart beat.

y coordination: volts (output of piezoelectric sensor)
x coordination: sampling points

Figure 21 Raw data for heart rate measurement

y coordination: BPM (Beats Per Minute)
x coordination: measurements

Figure 22 Heart rate from raw data

 

CNN Training Result

Two different kinds of training dataset are used in this experiment. One is only the time-domain dataset, the other is the dataset which combines frequency-domain data. Figure 23 and figure 24 show the different categories of datasets that we used in the experiment.

Figure 23 The amount of training datasets

Figure 24 The amount of test datasets

According to the result of CNN training (figure 25 and figure 26) , it shows that the training process can achieve the target in a very short time and an early epoch. The accuracy rate even up to 99% in about the forth epoch. That means the data that collected from Smart Necklace can be used and can successfully be trained to classify data from different categories.

Figure 25 The result of CNN training (time-domain dataset only)

orange line : validation accuracy
blue line : training accuracy

Figure 26 CNN training history (time-domain dataset only)

Compared with the previous training result of time-domain dataset, it will need more epochs in order to achieve the same accuracy rate. Therefore, the dataset that only contains time-domain data will be used for future experiment.

Figure 27 The result of CNN training (time-domain + frequency-domain dataset)

orange line : validation accuracy
blue line : training accuracy

Figure 28 CNN training history (time-domain + frequency-domain dataset)

 

Predicted Result

The predicted result is very accurate when we used CNN model to training. The main reason that caused the high accuracy rate is because the data collected from Smart Necklace are quite different when wearer is doing different movements. There is a significant difference of possibility between different categories, but as we can see, the system cannot one hundred percent classifiy the coughing data. I assume that we should collect more data to improve the accuracy.

label 0: heart rate (static)
label 1: consumption of water
label 2: consumption of food
label 3: oral speech
label 4: coughing

Figure 29 Prediction Result

Figure 30 The predicted score of label 0 test dataset (heart rate/static)

Figure 31 The predicted score of label 1 test dataset (consumption of water)

Figure 32 The score of label 2 test dataset (consumption of food)

Figure 33 The predicted score of label 3 test dataset (oral speech)

Figure 34 The predicted score of label 4 test dataset (coughing)

Conclusão

Conclusion

In this study, Smart Necklace has been introduced to detect and monitor the health information of the person wearing the device. It successfully uses a non-intrusive method to collect data. According to the results, it shows that every movement will create a vibration and different movements have a distinct characteristic. After training this data using a CNN, the system can classify the data accurately.

Future Application

This device is especially useful in terms of helping the elderly. We are now entering the era of an aging society and the number of old people living alone is increasing every day. This device can send out a warning if the person is choking, as well as detect heartbeat, and give feedback in real time. It can also help care-takers know whether the elderly are drinking enough water or eating enough food. This device truly has a wide range of potential applications

Future Work

Since this device needs to send out a warning or give feedback in real time, Smart Necklace needs to connect to the Cloud or a smart phone. In addition, Smart Necklace will be wireless and future changes in its design will appeal to an even wider population. Its functions should also be improved, which means collecting more data and adding more categories, such as laughing, singing or checking sleep quality (I have now started to collect data, figure 35).

Figure 35 Picture that I was collecting sleeping data

This study is just the beginning of Smart Necklace, it is expected to be developed with other researchers, making it more commonly used in the future.

Sobre mim

Hello everyone, I am Wei-Ching Chen. I am from Taiwan.

Ever since I was little, I have long been enthusiastic about science and have traveled abroad to visit science museums and enrolled in local children’s camps. Throughout my middle and high school years, I participated in many science-related extracurricular activities.

Last year, I designed head impact sensors and studied how it could help determine which movements could lead to concussion. My research was successful and got me a second place in the engineering category at the 2018 Taiwan International Science Fair. This award also led me to represent Taiwan at the 2018 Canada World-wide Science Fair, the largest science fair in Canada, and got a gold medal in senior group. Recently, I was honored to be one of the guest speakers at the WinHEC, which is one of the three biggest annual conferences held by Microsoft. I was on behalf of the youngest generation of females who try to make a career in the tech world, sharing with the other prominent speakers my opinions on some issues.

Now, I am presenting my ideas to the world through this platform. It's been a great experience for me to learn from others, and I desire to make improvement and exploration every day.

Saúde e segurança

I did all the experiments at school and at home all by myself. Although there isn't any electric shot, lazer, infrared waves or sharp things in my experiment, I am still very careful when collecting data. I would empty the area where I was going to do the experiment and be sure no one would suddenly interupt me. A part of my study is body collecting data, and fortunately there are a lot of volunteers help me with the experiment. The movement I required them to do is something like drinking water, eating food, chatting with me and sit still and don't move. With the coughing experiment, I found one of my classmates who was coughing because of the changing weather. It is kind of hard to find a person who is coughing but not sick and because I can not forced others to cough due to this action may harm your throat, so it took me a long time to collect data. I believe I have followed the safety guidelines when designing Smart Necklace and doing experiments.

Bibliografia, referências e agradecimentos

Acknowledgements:

Class 1434 and all of my friends, assisted me on collecting data and became volunteers of my project. Without their help, I could not have collected such an amount of data. As volunteers, they also gave me some advice about what kind of data I can collect and also told me the feedback after using me Smart Necklace. They were also very supportive in helping me overcome the hardship of facing obstacles.

Teacher Steve Grad, assisted me on my presentations and my report books to make sure I didn't make any errors.

Professor Chen Yun-Nung, an Assistant Professor at the Department of Computer Science and Information Engineering of National Taiwan University, is my advisor. She gave me a lot of advice about how to use a CNN. She not only shared her own experiment about doing research, but also discussed how machine learning can be used in this project. After several times discussing with her, I learned many things, and I love doing scientific research now, more than ever.

 

References:

[1] Duman, R. S., Li, N., Liu, R. J., Duric, V., & Aghajanian, G. (2012, January). Signaling pathways underlying the rapid antidepressant actions of ketamine. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/21907221

[2] Eating Disorders. (n.d.). Retrieved from https://www.nimh.nih.gov/health/topics/eating-disorders/index.shtml

[3] Shaun K Riebl, B. (2013). The Hydration Equation: Update on Water Balance and Cognitive Performance. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4207053/

[4] Blau JN, e. (2004). Water-deprivation headache: a new headache with two variants. - PubMed - NCBI. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/14979888

[5] Heart arrhythmia. (2017, December 27). Retrieved from https://www.mayoclinic.org/diseases-conditions/heart-arrhythmia/symptoms-causes/syc-20350668

[6] Autism spectrum disorder (ASD). (n.d.). Retrieved from https://www.nhs.uk/conditions/autism/symptoms/

[7] Chronic cough. (2017, August 22). Retrieved from https://www.mayoclinic.org/diseases-conditions/chronic-cough/symptoms-causes/syc-20351575

[8] Piezoelectricity. (2018, December 09). Retrieved from https://en.wikipedia.org/wiki/Piezoelectricity

[9] Sampling (statistics). (2018, October 18). Retrieved from https://en.wikipedia.org/wiki/Sampling_(statistics)

[10] Fast Fourier transform. (2018, December 06). Retrieved from https://en.wikipedia.org/wiki/Fast_Fourier_transform

[11] What is machine learning (ML)? (n.d.). Retrieved from https://searchenterpriseai.techtarget.com/definition/machine-learning-ML

[12] Prabhu. (2018, March 04). Understanding of Convolutional Neural Network (CNN) - Deep Learning. Retrieved from https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148

[13] Window function. (2018, December 04). Retrieved from https://en.wikipedia.org/wiki/Window_function