i-Cane: SMART NAVIGATION FOR THE VISUALLY IMPAIRED

Summary

The World Health Organization (2017) states that globally, approximately 253 million people live with some form of vision impairment. Visual impairment limits the ability to perform everyday tasks and adversely affect the quality of life, ability to interact with the surrounding world, thus discouraging individuals navigating unknown environments.

The goal of the project was to develop a simple intelligent cane (i-Cane), to help the visually impaired person to navigate their environment. The hypothesis was if different objects are placed at different distances in front of the i-Cane, then the device will be able to measure the distance of the object, classify the object and synthesize a voice alert warning the person.

A Raspberry Pi with a camera and an ultrasonic sensor was configured and a software that uses Google Cloud Vision API to identify objects and Festival Speech Synthesis software to generate voice feedback was developed.

Research included understanding and learning:

  1. Raspberry Pi
  2. Python programming
  3. Computer Vision software, including Google Cloud Vision API
  4. Leveraging the Pi Camera to take images 
  5. Speech Synthesis software that can communicate with the visually impaired person.

The project conceptualized a simple device, i-Cane, for visually impaired individuals that can warn visually impaired persons of approaching objects as they move around.

The data from the experimentation supports the hypothesis of the experiment.

The scope of the project can be expanded by adding multiple ultrasonic sensors and infrared sensors to detect approaching obstacles all around (360 degrees) the visually impaired person.

Question / Proposal

Question

What effect does the different obstacles at different distances have on the operation of the i-Cane device comprising of Raspberry Pi 3 Computing Platform, Ultrasonic Sensor, Pi Camera, and Python Program that interfaces with Ultrasonic Senor, Pi Camera, Google Cloud Vision API, and Festival Speech Synthesizing module?

Hypothesis

If different objects are placed at different distances in front of the i-Cane device comprising of Raspberry Pi 3 Computing Platform, Ultrasonic Sensor, Pi Camera, Python Program, Google Cloud Vision API and Festival Speech Synthesizer, then the i-Cane device will be able to measure the distance, classify the objects and synthesize a voice alert stating the object type and distance of object from the i-Cane device.

Flow

The project included designing and developing an intelligent cane (i-Cane) to assist a visually impaired person to navigate around his/her environment. The main process flow of the i-Cane device developed using Raspberry Pi Computing Platform and Python based software is as follows:

a.    Detect an object along the path of a visually impaired person using Ultrasonic Sensor and the object’s distance from the person using the speed of sound.  

b.    Use Pi Camera to take an picture of the object and use Google Visual Recognition (Vision API) to identify the object

c.    Finally, synthesize a voice/speech alert for the visually impaired person in natural language using Festival Speech Synthesizing module.

Research

Research included understanding and learning:

  • Visually impairment and its impacts on daily life
  • Raspberry Pi
  • Leveraging the Pi Camera to take images 
  • Ultrasonic sensor to detect objects
  • Python programming
  • Computer Vision software, including Google Cloud Vision API
  • Speech Synthesis software that can communicate with the visually impaired person.

The World Health Organization (2018) stated that globally, approximately 1.3 billion people live with some form of vision impairment[1][2]. Visual impairment limits the ability to perform everyday tasks and adversely affect the quality of life, ability to interact with the surrounding world, thus discouraging individuals navigating unknown environments[8].

The application of intelligent machines is transforming businesses and quality of life in every scope of life [8]. Raspberry Pi computer platform presents an endless opportunity to create many impactful projects [6]. Books on Pi[3][5] provided detailed information on the configuring the Raspberry PI and described ways to leverage the Pi for computing projects. Books[4] and websites[7] described in the great detail how to get started with Python programming-based software development for Pi. Pi along with Python-based software leveraging sensors like Pi Camera, Ultrasonic Sensor provides a low-cost computational platform for building intelligent machines. Participation in several summer camps focused on robotics, physics and electric circuit design have assisted me to gain knowledge about Raspberry Pi, Arduino, and Python Program. The websites on Pi Camera covered detailed steps on how-to connect Pi Camera with Pi and write software to take pictures using the camera [9][10].

Technologies like computer vision, speech synthesis, and speech recognition are at the center of intelligent machines that understand, learn, think and behave like a human. The critical portion of the research was to understand and identify the concepts of computer vision and speech synthesis. The project required the need to define an obstacle in front of the visually impaired person and how speech synthesis can be applied in the i-Cane device to communicate back to the person using natural language to alert the person with the type of the object and the distance of the object. Detailed research was conducted on available image recognition software such as Google Cloud Vision API, Amazon Image Rekognition, and Clarifai. The comparative study[17] showed that Google Cloud Vision is more accurate than Amazon Image Rekognition and Clarifai. Moreover, Google provided 1000 API calls per month free which I estimated to be sufficient for performing the experimentation. Research also indicated that setup and access of Google Cloud are simple and straightforward once one has a Google account. Research was also conducted on available speech synthesis software such as Festival Speech Synthesis, eSpeak,  and NeoSpeech. Festival provided a Python module and ran locally with no Cloud access required.

The project conceptualized a simple device, i-Cane, for visually impaired individuals that can alert visually impaired persons of approaching objects as they move around. The research did not indicate any existing work on a low-cost integrated solution for visually impaired although the research surfaced literature that described the usage of Ultrasonic Sensor and Pi camera standalone.

Method / Testing and Redesign

Independent Variables

  • The distance (relative location) between the i-Cane device and the obstacle
  • Different type of obstacles

Dependent Variables

Distance calculated using Ultrasonic Sensor and Python Program, Type of Object identified by Pi Camera and Google Cloud Vision API from Python Program, Voice alert generated by Festival Speech Synthesizing module and Python Program running on Raspberry Pi 3 Computing Platform.

Controlled Variables

  • Circuit assembly connecting the Raspberry Pi 3 to the Ultrasonic Sensor
  • Interface to Pi Camera
  • Python program – Design Criteria, Google Cloud Vision API, Festival Speech Synthesizing software

Step by Step Procedure

  • Gather materials
  • Setup the Raspberry Pi 3 and Pi Camera V2
  • Connect Ultrasonic Sensor to the Pi
  • Setup Google Cloud Access and Google Client on the Pi
  • Install Festival (Speech Synthesis Software) on the Pi
  • Develop the Python Program
  • Run the Experiment

In order to ensure experimentation process was fair and to have high confidence and credibility on the results, I repeated the experiment for multiple trials as follows:

  1. Ran trials using steps 3 – 6
  2. Marked a distance of 60 cm, 90 cm, 120 cm from the start line (where i-Cane  placed). During the experiment, the objects were placed on these marks and run the Python Program on Raspberry Pi
  3. In the first trial, placed the first object on the 60 cm mark.
  4. Placed the i-Cane on the start line in front of the object. Listened to voice alert and recorded the distance of the object and the type of the object as indicated in the audio.
  5. Repeated steps 4 and 5 for the first object placed at the 90 cm and 120 cm marks.
  6. Repeated steps 3 – 5 to run additional four trials by placing the remaining four objects at 60 cm, 90 cm, and 120 cm marks.

Software Development

Developed a Python program to do following four steps:

  1. Step 1: Wrote a Python function block, namely, detectObject, to trigger signal to Ultrasonic Sensor, and then waiting to receive the echo back from the sensor and calculate the distance between the ultrasonic sensor and the obstacle using the formula
    • S = 2 D / t i.e.  D = (S x t)/2 where,
    • S is Speed of sound and S = 34030 cm/s
    • D is Distance between an obstacle and sensor
    • t is Time taken by the sensor
  2. Wrote a Python function blocks
    • Step 2: to take photo/image using Pi Camera (takePhoto)
    • Step 3: to call Google Cloud Vision API to classify the image (classifyImage)
    • Step 4: to synthesize speech alert using Festival speech synthesize (synthesizeVoiceAlert)
  3. Wrote a Python main function block to call detectObject, and if the distance of an object is less than 150 cm, called takePhoto, classifyImage, synthesizeVoiceAlert

Design Criteria

  • Distance  < 150 cm, No voice alert synthesized
  • Distance  >= 150 cm, Voice alert made stating the object found and the distance from the i-Cane

I conducted the experiment in my home. There was no risk or safety issues as the device was operated by 5V battery pack.

Results

Data collected below was synthesized by i-Cane as voice by Festival Speech Synthesizer software component installed on Raspberry Pi and was confirmed using headphones

Distance Measurement Using Ultraviolet Sensor, Object Identification by Pi Camera and Google Cloud Vision API, Voice Alert by Festival Speech Synthesizer

Location

Number for Obstacles

Trial 1

(Instrument Case)

Trial 2

(Chair)

Trial 3

(Light/Lamp)

Trial 4

(Stepping Tool)

Trial 5

(Wall)

Location 1 (60 cm from i-Cane) Detected an object of type bag, gadget, handbag, technology, leather at a distance of 55.3 centimeters Detected an object of type of property furniture, floor, room, flooring at a distance of 55.35 centimeters Detected an object of type ceiling, light fixture, lighting, lamp at a distance of 59.28 centimeters Detected an object of type blue furniture, table, floor, flooring at a distance of 59.86 centimeters Detected an object of type ceiling, wall, floor molding wood at a distance of 59.47 centimeters
Location 2 (90 cm from i-Cane) Detected an object of type black floor, flooring, bag, furniture at a distance of 85.57 centimeters Detected an object of type of property room, furniture, house, floor at a distance of 85.77 centimeters

Detected an object of type light fixture, lighting, lamp, ceiling, wood at a distance of 88.75 centimeters

Detected an object of type blue furniture, room, floor, flooring at a distance of 82.3 centimeters Detected an object of type ceiling, wall, floor, automotive, exterior daylighting at a distance of 88.92 centimeters
Location 3 (120 cm from i-Cane) Detected an object of type of white floor, flooring, gadget, wood at a distance of 115.41 centimeters

Detected an object of type property room, furniture, floor, house at a distance of 111.98 centimeters

Detected an object of type white house, room, floor, light at a distance of 119.1 centimeters Detected an object of type floor, room, flooring, furniture, wall at a distance of 109.88 centimeters Detected an object of type room, wall, floor, product, furniture at a distance of 120.96 centimeters

Location 1: distance of the object from i-Cane measured by Ultrasonic Sensor was with in 5 cm i.e. 8.33% of actual distance (60 cm).

Location 2: distance of the object from i-Cane measured by Ultrasonic Sensor was with in 5 cm i.e. 5.55% of actual distance (90 cm)

Location 3: distance of the object from i-Cane measured by Ultrasonic Sensor was with in 10 cm i.e. 8.33% of actual distance (120 cm)

Trial 1: Insturment case was identified as bag 2 out of 3 times.

Trial 2: Chair was identified as furniture 3 out of 3 times.

Trial 3: Lamp/Light was identified as light or lamp 3 out of 3 times.

Trial 4: Stepping tool was identified as furniture 3 out of 3 times.

Trial 5: Wall was identified as wall 3 out of 3 times.

 

Video of the experimentaion is enclosed herein.

Conclusion

The hypothesis of the experiment was that if different objects are placed at different distances in front of the i-Cane device comprising of Raspberry Pi 3 Computing Platform, Ultrasonic Sensor, Pi Camera, Python Program, Google Cloud Vision API and Festival Speech Synthesizer, then the i-Cane device will be able to measure the distance, identify the type of the objects and be able to synthesize a voice alert stating the type of the object and distance of object from the i-Cane device.

The data from the experimentation supports the hypothesis of the experiment.

The experiment demonstrated that as an obstacle is placed different location (at 60 cm, 90 cm, and 120 cm distance) in front of the i-Cane device the i-Cane built using Raspberry Pi 3 computing platform and Python programming is able to calculate the distance of the obstacle from the i-Cane using Ultrasonic Sensor, able to identify the type of object using Pi Camera and Google Cloud Vision API, and synthesize voice alerts using Festival Speech Synthesizer indicate the type and distance of the obstacle. 

isually impaired individuals may be discouraged from moving freely and comfortably. This project establishes a very simplified intelligent cane prototype for visually impaired individuals. This i-Cane will assist someone without vision to be able to identify obstacles and gauge its distance when they are moving in a house, or maybe walking in a mall or a grocery store. 

TThe scope of the project can be expanded by adding multiple Ultrasonic Sensors, IR sensors and Pi Camera to detect approaching obstacles all around (360o) the visually impaired person. That would require extending the electric circuitry and the Python program. This solution and especially visual recognition part can even be extended to help visually impaired individuals to read and see things around in their environment for example label on medicine buttons, nutrition facts on food packages, menu in the restaurant, etc.

The successful working prototype of intelligent cane (i-Cane) or smart walking stick that uses Raspberry Pi 3 Computing Platform, Ultrasonic Sensor, Pi Camera,  Python Program, Google Cloud Vision API and Festival Speech Synthesizer to synthesize voice alert indicating the type and distance of obstacle approaching in the path of a visually impaired represents how simple application using engineering, science, and technology can help improve the quality of life for the visually impaired individual.

About me

My name is Arko Ghosh. I am a 9th grader at King High School in Tampa, Florida.  I have three things that I am passionate about - robotics, soccer and playing the trumpet. I am the Captain and Engineering Lead for the FIRST Tech Challenge (FTC) King High School team, Hadron Knights. As the Engineering Lead, I am responsible for designing and building the robot that can perform the tasks for the challenge. I enjoy playing soccer and play for the King High School team and the Tampa Bay Chargers soccer club. I love playing the trumpet in the high school marching band and the jazz club. I participate in several school clubs including the Mu Alpha Theta math club, Speech & Debate, Science National Honors Society, Technology Student Association, Firsthanded 3D Printing and Future Business Leaders of America. I want to pursue a career in computer science and electrical engineering, concentrating on mechatronics.

My great grandmother lost her vision when she was in her 40s. When she passed away, she was ninety-four. For fifty years, she lived in the darkness of her world, never seeing any of her grandchildren or great-grandchildren. Her life was restricted to one room in her house and dependent on someone to assist her with her daily chores. She loved her garden but never saw her rose blooms.  She and others who are visually challenged inspire me for this project.

The innovative work of Mr. Bill Gates and Mr. Dean Kamen inspires me.

Health & Safety

The experiment was conducted in the home and the experiment setup only needed a 5V DC supply. There was no health and safety issues.

The project didn't require any hazardous material or didn't require any interaction with live specimens.

Bibliography, references, and acknowledgements

  1. Global Data On Visual Impairments 2010. World Health Organization, 2012, Global Data On Visual Impairments 2010, www.who.int/blindness/GLOBALDATAFINALforweb.pdf.
  2. “Global Data on Visual Impairment.” World Health Organization, World Health Organization, 8 Dec. 2017, www.who.int/blindness/publications/globaldata/en/.
  3. Halfacree, and Upton. Raspberry Pi User's Guide. N.p.: Wiley, 2012. Print.
  4. Monk, Simon. Programming the Raspberry Pi, Second Edition: Getting Started with Python. N.p.: McGraw Hill Education, 2015. Print.
  5.  McManus, Sean Raspberry Pi for Dummies 2nd Edition; N.p.: Wiley, 2014. Print
  6. Raspberry Pi - Teach, Learn, and Make with Raspberry Pi, Raspberry Pi Home Comments. N.p., n.d. Web. 26 Oct. 2015. https://www.raspberrypi.org/documentation/usage/demos/README.md
  7. Python 2.7.10. Program documentation. Python.org. 23 May, 2015. Web. 26 Oct. 2015. https://www.python.org/downloads/source/
  8. The NFB Center of Excellence in Nonvisual Access to Education, Public Information, and Commerce (CENA)." Technology Center. The Journal of Blindness Innovation and Research, n.d. Web. 26 Oct. 2015.
  9. "Camera Module - Raspberry Pi." Raspberry Pi. N.p., n.d. Web. 10 Oct. 2016.  https://www.raspberrypi.org/products/camera-module-v2/
  10. "Getting Started with Picamera." Getting Started with Picamera | Raspberry Pi Learning Resources. N.p., n.d. Web. 15 Oct. 2016. https://www.raspberrypi.org/learning/getting-started-with-picamera
  11. Python 2.7+ Installation." 1. Python 2.7+ Installation — Picamera 1.10 Documentation. N.p., n.d. Web. 20 Oct. 2016. http://picamera.readthedocs.io/en/release-1.10/install2.html
  12. "Image Recognition." Image Recognition - MATLAB & Simulink. N.p., n.d. Web. 22 Oct. 2016. https://www.mathworks.com/discovery/image-recognition.html
  13. "Amazon Rekognition – Deep Learning-based Image Analysis." Amazon Web Services, Inc. N.p., n.d. Web. 25 Oct. 2016. https://aws.amazon.com/rekognition/
  14. "Visual Search." Search Visual Content by Similarity or Tag | Clarifai. N.p., n.d. Web. 25 Oct. 2016. https://clarifai.com/visual-search
  15. "Watson Developer Cloud." Visual Recognition | IBM Watson Developer Cloud. N.p., n.d. Web. 25 Oct. 2016. https://www.ibm.com/watson/developercloud/visual-recognition.html
  16. "Vision API - Image Content Analysis | Google Cloud Platform." Google Cloud Platform. N.p., n.d. Web. 29 Oct. 2016.  https://cloud.google.com/vision/
  17. Domes, Scott. “We Compared the 3 Best Image Analysis API's - Here's What We Learned.” We Compared the 3 Best Image Analysis API’s — Here’s What We Learned, Code == Life, 6 Mar. 2017, engineering.musefind.com/we-compared-the-3-best-image-analysis-apis-here-s-what-we-learned-2d54cff5ae62
  18. "Quickstart | Google Cloud Vision API Documentation | Google Cloud Platform." Google Cloud Platform. N.p., n.d. Web. 01 Dec. 2016. https://cloud.google.com/vision/docs/quickstart
  19. Authenticating to a Cloud API Service | Google Cloud Vision API Documentation |Google Cloud Platform." Google Cloud Platform. N.p., n.d. Web. 30 Oct. 2016. https://cloud.google.com/vision/docs/common/auth#using-api-manager
  20. "What Is Speech Synthesis? - Definition from WhatIs.com." WhatIs.com. N.p., n.d. Web. 10 Oct. 2016. http://whatis.techtarget.com/definition/speech-synthesis
  21. "ESpeak: Speech Synthesis." SourceForge. N.p., n.d. Web. 30 Oct. 2016. http://espeak.sourceforge.net/
  22. "The Festival Speech Synthesis System." Festival. N.p., n.d. Web. 30 Oct. 2016. http://www.cstr.ed.ac.uk/projects/festival/
  23. "NeoSpeech Text-to-Speech Everything You Need to Know about Text-to-Speech Software and Natural Text-to-Speech Voices." NeoSpeech Text-to-Speech Software & Synthesized Voices. N.p., n.d. Web. 10 Oct. 2016. http://neospeech.com/
  24. "Speech Synthesis on the Raspberry Pi." The Festival Speech Package | Speech Synthesis on the Raspberry Pi | Adafruit Learning System. N.p., n.d. Web. 01 Dec. 2016. https://learn.adafruit
  25. "Acapela Group - Voice Synthesis - Text to Speech | Voice Solutions." Acapela Group Voice Synthesis Text to Speech. N.p., n.d. Web. 30 Oct. 2016. http://www.acapela-group.com/