UMaine Partnering in Grant to Engineer Navigation Application for Blind and Visually Impaired
Contact: Kate Beard, (207) 581-2147 or beard@spatial.maine.edu
Researchers in the UMaine department of spatial information science and engineering have received a four-year, $700,937 grant from the National Science Foundation to participate in an international project they hope will eventually result in a device that could help blind and visually impaired people lead more independent lives.
Kate Beard, the director of the National Center for Geographic Information at UMaine, and faculty members Nick Giudice and Reinhard Moratz will work with researchers in Philadelphia and Germany to develop computational methods for object detection, spatial scene reconstruction, and natural language spatial descriptions to describe prototypical indoor spaces such as rooms and offices.
Development of those computational methods could lead to an application that could provide a blind or visually impaired person with enough information to navigate a space with which he or she is unfamiliar.
An example scenario would be a blind or visually impaired person who enters a doctor’s office, which the person has not previously visited. The person would enter the space and use the device to take a series of pictures. The pictures would be processed and then translated verbally to the user, who would be able to query the program for locations of, for example, the registration desk, the closest empty chair in the waiting room, or the restroom.
Another application could be to find misplaced objects in a room – say, for example, a person puts down a coffee cup but is unable to find it again. If a representation of the room exists from processing many previous pictures, the device could search for the object based in part by determining what in a room was different than it was before.
Beard said the ultimate goal of the project would be an application on a device such as a cell phone.
“I don’t know if in the course of the grant we would have a fully working application on a phone as the footprint, but hopefully we would have prototype pieces of it working,” Beard said. “There are some phone applications now that do fast-image processing on a phone, but this application would require another level to locate objects in space and convey the objects and aspects of that space verbally to the user.”
UMaine’s NSF grant is part of $1.2 million in funding attached to the project. The other institutions involved are Temple University and University of Pennsylvania, which are both in Philadelphia, and the University of Bremen in Germany.
Beard, Giudice and Moratz are specifically researching ways in which spaces can be efficiently represented for users to query and receive verbal responses.
The UMaine researchers will also do human subject experiments to determine both the standard and different linguistic phrases people use in order to describe spaces.
“There will be a sequence of these experiments to help us first build up the features we should be looking for, the key things that we need to extract from the images, and then also what makes sense in terms of vocabulary and phrasing,” Beard said.
Giudice, assistant professor in the department of spatial information science and engineering and the director of UMaine’s Virtual Environments and Multimodal Interaction (VEMI) Lab, will be a key to the project, Beard said.
Giudice is himself visually impaired, and will be able to provide real-life assistance with verbalizing navigation.
“Nick thinks this application would be really helpful,” Beard said. “The first time you go to a new place, there’s always that uncertainty. If there was a quick way of getting some orientation in that space, he feels it would give blind or visually impaired people a greater sense of independence, because they would feel more comfortable going into an unfamiliar space.”
The other key piece of the UMaine research will be building up representations of spaces. Beard and her team are planning to use ontologies, a scientific term for a framework of concepts and relationships among them, to support the image processing, reasoning about a space, and generating natural, language-like expressions.
“You could think of a room ontology as a computational model of a room,” Beard said. “The basic representation includes a floor, ceiling, certain number of walls. Then as the images are processed this basic room representation becomes more specific, such as a specific wall has certain number of windows or a door. So it goes from a generic box to a box that has been reconfigured based on the information from the images.”
The researchers would also have to determine the best way to convey distances, directions and positions.
Researchers at the University of Bremen have been working on spatial linguistics models using ontologies. Those researchers will collaborate with UMaine on extending the ontologies and spatial expressions that may be most useful to blind users.
At Temple and Penn, researchers are working on image sequencing, registration and feature detection. Among the challenges they face are determining how many pictures the user’s device would need in order to produce an accurate spatial reconstruction, and how to translate three-dimensional objects that may be overlapped with other objects – for example, a chair that is behind a desk.
The three projects together would begin to create a framework for an application in the future.
“What we envision is the experiments at the different locations will be linked together in building this model,” Beard said. “Our piece is doing the experiments and building these computational representations.”