eSight Eyewear and Smart Glasses from Assisted Vision

http://www.afb.org/afbpress/pub.asp?DocID=aw150506 During the past few years, the community of people with visual impairments has become increasingly excited over the accessibility prospects and possibilities of Google Glass, a pair of smart glasses that incorporates cameras, a heads-up display, and a live data connection to enhance the wearer’s ability to interact with his or her environment. What many are not as familiar with, however, are the great strides that have already been made by researchers focused on similar technologies and how they can be used to assist people with visual impairments to navigate and interact with their worlds on a footing more equal to that of their sighted contemporaries. This article discusses two such efforts: A pair of smart glasses under development by a team of researchers led by Oxford University’s Stephen Hicks, and the digital eyewear, available for purchase as of October 2013, from Ottawa-based eSight Corporation. eSight Corporation For nearly 30 years, Canadian electrical engineer Conrad Lewis has made a point of keeping up with all the latest access technologies. His two sisters Julia and Anne were both diagnosed in their 20s with Stargardt Disease, an early onset form of macular degeneration. Prompted by their diagnoses, Lewis–who began his professional career as a business executive and is now a venture investor–began to bring home new gadgets and pieces of access software he’d come across at trade shows and through his growing network of professional connections. In the middle of the last decade Lewis took note of the growing convergence of mobile processing power and lightweight, high-resolution video displays. Perhaps he could leverage this coming convergence into a workable product that would enable his sisters to use their limited eyesight more effectively. In 2007, Lewis founded eSight Corporation with the help of US and Canadian angel investors, along with grants from various foundations and government agencies. “Others had previously worked on head-mounted displays for the visually impaired, but they were too large and heavy, and didn’t allow people to be mobile–not at all what Conrad had in mind,” says Kevin Rankin, president and CEO of eSight Corporation, where Conrad Lewis currently serves as Chairman of the Board. The device Lewis envisioned would also require much faster image processing than was available at the time of the company’s founding. So he and his team of engineers set about writing and optimizing software, testing and customizing components, and building prototypes for two generations of eSight glasses. They completed their first pre-production model in mid-2012, and in October of 2013 began offering their eSight glasses for sale in the US and Canada. eSight Glasses: How They Work eSight glasses are about the size of a pair of wraparound sunglasses. They enable a user to magnify and view objects as close as 12 inches away and as far away as an object across the room, across the street, or across a field. A high-resolution video camera with zoom capabilities is built into the bridge, and a cable runs from one of the earpieces down to a hip-carried processing unit and power source. “The glasses are custom made using lenses ground to the wearer’s own prescription,” says Rankin. “These lenses are then overlaid with a transparent OLED (organic light emitting diode) display that can be user adjusted to fill their entire field of view, or just the upper portion, while allowing use of peripheral vision and awareness, and most importantly mobility.” Think of a pair of bifocals, where the user can choose between magnified and contrast enhanced or their regular vision for any activity of daily living, depending on whether he or she focuses his or her gaze through the upper or lower half of the lenses. The eSight camera captures what’s ahead and sends it to the processing unit, which is about the size of a large-screen smartphone and about twice as thick. There the images are processed frame by frame in real time. “The unit allows the user to adapt to their personal preferences and needs with two easy-to-use dial controls, including an up to fourteen times zoom, contrast, and various color adjustments to make the real-time image easier to see and enjoy,” Rankin explains. Lewis did not want his sisters and other users to have to constantly switch back and forth between their standard prescription glasses and their eSight digital eyewear. “The way we designed them, a wearer could rely on their own prescription lenses to navigate their living room or other familiar surroundings, switch to half screen mode with some magnification and enhanced contrast to watch television, or choose a full-magnification, full-screen mode to read a book with white letters on a black background,” he says, adding, “eSight users are now sharing amazing stories of actually seeing all of the important details while shopping, walking through airports, being at school, and at work.” One User’s Perspective The eSight glasses went on sale last October, priced at $14,950. One of the first purchasers was Yvonne Felix, who lives with her husband and their two young sons in Hamilton, Ontario. Yvonne was diagnosed at age 7 with Stargardt Disease. By 15 she could no longer see the drawings filled with fairies and unicorns she loved to create. “I’d have to finish them in a single sitting,” she recalls. “Otherwise I’d lose my place.” In high school teachers would not allow Yvonne to attend art class because they didn’t know how to grade her work. They also discouraged her from assembling a portfolio and applying to art college. When Yvonne was 25 she applied, anyway, was accepted, and after graduation she became a community artist with two public installations to her credit–a public conversation area and a large magnifying glass that’s also a sundial. Yvonne read about the glasses in a Foundation Fighting Blindness newsletter, and purchased a pair with the help of a generous private donor and several public fundraisers. “They brought them to my home to try,” she relates. Yvonne did not wear prescription lenses, so her first test was using full screen magnification. The results were startling. “The very first thing I saw was my husband and my boys,” she remembers. “They were beautiful. They looked just like I had always imagined.” At the time, Yvonne was completing a painting for a charity auction–an abstract depicting her blind spot. “When I saw it through the glasses, I wanted to redo the entire canvas,” she says. “My mind’s eye and my new eyes had a lot of getting to know each other to do.” Yvonne’s vision was improved even more with the addition of prescription lenses. “Sometimes it’s like my blind spot isn’t even there, anymore,” she says. “I can see the dials on the oven, and these days when the house gets dirty I notice it, which is a mixed blessing.” Yvonne’s brother, William, also has Stargardt disease, and he is in the process of getting a pair of eSight glasses for himself, too. Her elder son, Noah, has also tried on Yvonne’s glasses. “To him it’s like a magic trick that lets me read print books to him at night,” she says. Smart Glasses from Assisted Vision The benefits of eSight eyewear are limited mostly to people with partial sight between 20/60 and 20/400. This leaves out a considerable swath of individuals who have much lower visual acuity. Happily, a small team of British researchers led by Stephen Hicks, PhD, Research Fellow in Visual Prosthetics in the Nuffield Department of Clinical Neuroscience at the University of Oxford, are well on their way to producing a different kind of device that could assist individuals with useable vision, less than 20/400, to identify objects and more safely explore and navigate their environment. How Assisted Vision Smart Glasses Work Like Conrad Lewis, Hicks also saw the benefit of working with off-the-shelf technology. “It occurred to me an excellent starting point might be to pair object recognition software with a heads-up display,” he explains. In 2010 Hicks began working with LabVIEW, object recognition software from National Instruments, which later gave him an award for innovative use of their product. “Traffic signs were fairly easy to recognize, and they were also easy to set up in a lab,” he says. Hicks created signs the size of CD jewel cases and hung them on a wall about 4 meters–a bit more than 13 feet–from several individuals with vision less than 20/600. “Without enhancement, none of the subjects could pick out the signs,” he recounts. “We trained a video camera on the wall and used the object recognition software to spot the sign. The image was processed and enhanced, then projected onto the heads-up display of a gaming helmet. Every one of the subjects could now see a patch of brightness in the direction of the wall where the sign was located.” Hick’s proof of concept model used a single video camera, so there was no way to distinguish distant from nearby objects. In a happy happenstance, however, it was about that time when Microsoft introduced Kinect, a gaming device that creates a 3D map of a room and identifies game players, tracking their movements and gestures. Kinect uses a single video camera, but it also projects thousands of tiny infrared dots, and uses their reflections and a complex set of algorithms to calculate depth, much like radar or sonar. “With Kinect we could create a 3D map of objects up to 20 feet away,” says Hicks. “But we now had too much information. We not only had to identify objects, we had to figure out which objects were important and which were just background.” Hicks and his team solved the problem by taking a giant step backwards. “We stopped trying to identify the objects,” he explains. “Instead of trying to pick out that table three feet ahead and tagging it as a table, we began simply presenting that table as an area of brightness, the closer the brighter. Hicks also simplified the image by removing the far?away back half, and adding enhanced contrast controls. “Often all it takes is a tiny hint of where something fairly close is located to find a door or orient yourself inside a room,” he says. Hicks and his team continued to refine and improve their smart glasses. “We didn’t want to replace anyone’s usable vision, we wanted to enhance it,” Hicks says. Toward that end they assembled a different sort of heads-up display using even more off-the-shelf technology. This new display projected the visual enhancements onto a transparent OLED screen. The wearer can use as much of his or her remaining sight as possible to identify that table, helped along by the device’s brightness, contrast and edge enhancements. “Hold a hand in front of your face and the image would show through the glasses, but with an aura of brightness at the edges to help identify it,” Hicks explains. One User’s Story In August of 2013, University of London lecturer in French Dr. Hannah Thompson spent two hours with Hicks and his team testing out the glasses. “When I put them on I felt like a character in a science fiction novel, she relates in a blog post. “I was suddenly seeing the world in a completely different way. Objects which would have been impossible for me to see shone before my eyes in shades of pink and white. “I found the glasses incredibly easy to use, and within minutes I was happily navigating my way around a series of obstacles. I would find these glasses especially useful at night, in glaring sunlight or in dappled shade. They would not only stop me from walking into things, they would also help me keep a watchful eye on my children, who are often the first things to disappear when light conditions affect my vision.” Indeed, light conditions are one of the few remaining hurdles Hicks and his team must overcome before they turn the device over to the engineers to miniaturize the components and incorporate them into an attractive and comfortable pair of eyeglass frames. “The infrared dots work well inside to fix position, but as soon as you step out into bright sunlight they wash out and become increasingly useless,” Hicks explains. Hicks has engaged a British camera company to create an imaging processing unit that works in bright light to generate real time 3-D maps and still be small enough to fit on the bridge of a pair of glasses. “We could actually perform all of the processing on the glasses themselves,” he adds, “but we will still need to use a separate power supply, because adding a battery would make the smart glasses too heavy for comfort.” Hicks has many other enhancements planned for the near future. “We’ve circled back around to our starting point with image recognition, which we could use to identify faces, signs, even headline text. Unfortunately, we can’t use color as markers for identified objects, because many persons with extremely limited vision have lost their color perception. We could create blinking patterns, however, or play sound cues through headphones–perhaps bone conducting headphones so we don’t interfere with environmental sound cues.” Hicks is currently in the final months of a four-year pilot study, and by the end of 2014 he is hoping to begin manufacturing and marketing his glasses through a startup company named Assisted Vision for approximately]]>