Skip to content
Search
Search
 

Robert Margolis, 79, settles into his chair in the corner of the living room at his home in Pittsburgh. With the tap of a stylus on the touchscreen of a tablet computer, he begins scrolling through a succession of digital pictures and sound recordings.

He looks at a photograph of a familiar gateway and squints. Then his face lights up as he recalls a trip to the Pittsburgh Zoo three years earlier. "Oh sure, I remember this, I really do," he tells his wife, Betty. "I remember we went with Lynn," the couple's daughter.

This isn't just any digital photo show, or your typical trip down memory lane. Robert Margolis several years ago was diagnosed with mild cognitive impairment, which can be a precursor to Alzheimer's disease, a neurological condition that snuffs out recent memories. As the condition intensifies, it eventually cuts people off from normal life and from their loved ones. A retired pharmacist, Margolis can talk lucidly about the spread of national chain drugstores. And he can provide some details about structured parts of his life like his singing group or a class he's taking on Irving Berlin. But he probably can't say what he did yesterday.

He held on to his zoo trip memories, however, with help from the high-tech memory reinforcement system — MemeXerciser — he's holding in his lap. Margolis was involved with early testing of the system currently in development at Carnegie Mellon University's Human-Computer Interaction Institute; the device is still a few years away from hitting the market.

Computerized caregivers

MemeXerciser is one example of an emerging class of intelligent devices meant to provide support for people with cognitive decline from Alzheimer's and other conditions associated with aging. Because memory loss may be the most devastating effect of such disorders, it's what most of these early intelligent assistive systems focus on. Some like MemeXerciser are all about helping preserve memory. Others provide practical, on-the-fly support in social or other situations where a little nudge from a context-aware computerized helper can smooth over momentary lapses.

These forthcoming devices aren't meant as electronic replacements for caregivers or as robotic companions. Robots for moving people with disabilities around, bathing them or providing artificial social interaction are also being studied — in certain cases even introduced — by governments of some Asian and European countries. Many mimic human or animal forms.

But MemeXerciser, says its developer, Carnegie Mellon doctoral candidate Matt Lee, is the opposite of a personified robot. It's designed to be as transparent as possible and give only the help needed. "Caregivers often give little hints, and we wanted to replicate that," he says. "People feel good when they can remember something for themselves."

Memory prompts

The system uses what he calls a "scaffolding" of assistance. While looking at zoo pictures, Margolis tries to remember details about a snack break. Unable, he taps the computer screen again, and the device plays a sound clip from that moment three years ago. Still struggling, he taps again, and his wife's recorded voice explains the situation — there was no cake left, so he settled for yogurt. "It was good though," he says agreeably.

To use the system, a person with memory loss wears a small Microsoft-built camera that automatically takes hundreds of pictures during a selected activity — a party, a wedding, a day trip. Computer intelligence comes into play during picture selection. If caregivers like Margolis' wife had to sort through hundreds of images for every event, the system wouldn't get used, Lee says. But by employing a sophisticated algorithm, MemeXerciser accurately guesses pictures likely to be important. For instance, the camera's built-in GPS sensor might indicate Margolis lingered long near the elephant enclosure. Voila, a few elephant pictures are chosen.

Betty Margolis makes a final selection — no computer can tell for sure what is meaningful to a person — and adds spoken commentary. With perhaps 10 key images selected, the whole process might take a few minutes, rather than the hours it would take to go through all photos. Caregivers review the electronic slideshow regularly soon after the event, Lee says, and hopefully important memories will be preserved even years later, rather than allowed to slip away.

Social assistance

What if, instead of intensively exercising a failing memory, someone still living independently just wants a prompt now and then?

"If you ask older people what most disturbs them, it's not things like forgetting medication or getting lost," says Ronald M. Baecker, a University of Toronto computer scientist who heads the school's Technologies for Aging Gracefully (TAG) Laboratory. "Several studies show it's forgetting names."

To help people remember names, TAG Lab researchers developed Fried Forecaster, which runs on a cellphone and uses GPS to track your location. It predicts, based on information you've previously entered about your social network and places you go, who you're likely to meet. It's similar in some ways, Baecker says, to phone-based GPS systems teenagers use to keep tabs on networks of friends.

The TAG Lab is working on a similar system called Marco Polo, which helps people with nominal aphasia. This condition, often caused by a stroke, can make it extremely difficult to recall words or names. But Marco Polo, an application for iPhones or Android devices, is designed to take up the linguistic slack. "Based on where you are, it tells you words or phrases that might be useful," he says. So if you frequently buy the same loaf of bread at the same bakery, the device infers what you're doing and gives you the words, if needed. "You can look at it yourself for a prompt, or show it to someone, or even have it talk to them."

Marco Polo, in its third iteration, is on the verge of being ready to market, Baecker says.

In the Robotics Institute at Carnegie Mellon, roboticist Takeo Kanade is developing a memory-support system that's less a device you carry with you to help out, à la the TAG Lab systems, and more of a computerized extension of your own body.

First Person Vision, as it's called, uses two tiny cameras mounted on eyeglasses. One camera looks at your eye to see where you're looking, and another looks at whatever you are observing. Then, based on what you or a helper has configured the system to do, it provides information. If you want people's names, for instance, it could use a face-recognition program to identify people you look at, providing details through a holograph projected in front of your eyes.

It isn't the first concept of a wearable device to provide information, says Kanade, director of Carnegie Mellon's Quality of Life Technology Center. In the past, however, developers were intent on making such systems autonomous, needing little or no input from users. They understand now that computers simply aren't yet capable of some things needed for interacting with people — even something as basic as distinguishing men from women, he says. "Then, we didn't realize people could help the computer do a better job," he says. "Now we see it as a symbiotic system, human and computer working together."

A monitor for daily routines

Not every example of intelligent assistive technology is about memory, however.

Anind K. Dey, a Carnegie Mellon professor of human-computer interaction, is collaborating on the development of a GPS-based system for older drivers that determines how they like to drive, and chooses appropriate routes.

"We know elder drivers typically don't like making unprotected left turns, and some will drive a long way and make three right turns to avoid a left turn," he says. So the system automatically chooses efficient routes drivers are likely to be comfortable with. It can even predict where a driver is headed and help them smoothly avoid road hazards and traffic jams.

Matt Lee, Dey's student and developer of the MemeXerciser, is working on another system that relies on sensors embedded in everyday devices that could provide early warning of cognitive decline. Sensors in a pill box, for instance, monitor whether a person is taking the right medicine at the right time. Lee has outfitted a coffeemaker with sensors as well. "Before you actually reach a point you can't make a pot of coffee, the sensors in a smart coffeemaker can show you're starting to make more frequent mistakes," he says.

In some ways, the system mimics observations that a professional therapist might make. But someone first has to be suitably alarmed to call one in, after someone forgot to take his or her medication, for example. However, long-term data recorded by the MemeXerciser might reveal patterns too subtle to be noticed by a loved one, and treatment can start before a problem becomes advanced.

These intelligent systems aren't designed to replace the caregivers, spouses or family members of those with cognitive decline, but they may well help lighten the load, Lee says.

Betty Margolis admits constant questions from a forgetful spouse can be difficult. And not being able to share memories with her husband is painful. She says MemeXerciser and devices like it can't reach the market soon enough.

"This is a very helpful gadget — you can see it really works," she says. "I just wish we had it with us all the time."

Chris Carroll lives in Maryland.