To find the best memory cues for Mr. Reznick’s experiences, the researchers — Anind K. Dey, a computer science professor at Carnegie Mellon University, and Matthew Lee, a graduate student — considered the types of images that had proved the most effective in previous SenseCam studies.
They soon realized that the capriciousness of memory made answers elusive. For one subject, a donkey in the background of a barnyard photo brought back a flood of recollections. For another, an otherwise unremarkable landscape reminded the subject of a snowfall that had not been expected.
Still, the researchers came up with some broad rules for identifying and retrieving images likely to serve as memory triggers. For a people-based experience like a family reunion, the system selects photographs in which faces are clearly discernible; for a location-based experience like a visit to a museum, it uses geographical positions provided by GPS and accelerometer data to judge what images might be most salient — for example, when a subject might be hovering at one spot, like in front of a painting.
Research groups elsewhere are experimenting with other techniques to summarize and make use of SenseCam data. Alan Smeaton and colleagues at Dublin City University in Ireland are comparing images to categorize them by activity — shopping, for example — so the system can put together a visual summary of the day. At the University of Toronto, a group led by Ronald M. Baecker is investigating the usefulness of complementing SenseCam images with an audio narrative created by a loved one.
Once the system selects some photos from the hundreds taken, the caregiver winnows down the candidates, adding cues like audio from the voice recorder, verbal narration and brief text captions. The final product is a multimedia slide show on a tablet computer that allows the patient to dig deeper into highlighted parts of some images by tapping on the screen. The first tap plays audio, the second shows captions.
“The design is intended to give the patient the ability to engage actively with the experience instead of simply flipping through some pictures,” said Mr. Lee, the graduate student. Testing the system with the Reznicks and two other couples, he and Dr. Dey found that it helped patients recall events more vividly and with greater confidence than when they simply went through all of the images.