Total Recall: How the E-Memory Revolution Will Change Everything (29 page)

O’Conaire, C., N. O’Connor, A. F. Smeaton, and G. Jones. “Organizing a Daily Visual Diary Using Multi-Feature Clustering.” SPIE Electronic Imaging—Multimedia Content Access: Algorithms and Systems (EI121), San Jose, California, January 28-February 1, 2007.

Smeaton, A. F. “Content vs. Context for Multimedia Semantics: The Case of SenseCam Image Structuring.” SAMT 2006—
Proceedings of the First International Conference on Semantics and Digital Media Technology
. Lecture Notes in Computer Science (LNCS), Athens, Greece, December 6-8, 2006.

Smeaton, A. F., D. Diamond, and B. Smyth. “Computing and Material Sciences for LifeLogging.” Presented at the Memories for Life Network Workshop 2006, British Library Conference Centre, London, UK, December 11, 2006. Memories for Life Web site.

There is lots of other work on automatic summarization; for example, this paper on video summarization:

Sundaram, Hari, and Shih-Fu Chang. “Condensing Computable Scenes Using Visual Complexity and Film Syntax Analysis.” Second IEEE International Conference on Multimedia and Expo (ICME-2001), Tokyo, Japan, August 2001.

We mention a wearable system from the University of Tokyo that includes a brain wave sensor in a baseball cap. Their system also continuously captures video, GPS, gyroscope, and accelerometer data.

Aizawa, Kiyoharu, Datchakorn Tancharoen, Shinya Kawasaki, and Toshi hiko Yamasaki. “Efficient Retrieval of Life Log Based on Context and Content.”
Proceedings of the First ACM Workshop on Continuous Archival and Retrieval of Personal Experiences
(CARPE ’04), New York, October 15, 2004, 21-31.

Hori, Tetsuro, and Kiyohara Aizawa. “Context-Based Video Retrieval System for the Life-Log Applications.”
Proceedings of the 5th ACM SIGMM International Workshop on Multimedia Information Retrieval
, Berkeley, California, 2003.

reQall has a Web site, and there are a number of good articles about reQall, including the one in
Forbes
, below.

reQall Web site.
http://www.reQall.com

Woyke, Elizabeth. 2008. “You Must Remember This.”
Forbes
(September 30).

reQall cofounder Sunil Vemuri previously developed a PDA system to record audio and location. Calendar, e-mail, and common Web site and weather reports could also be captured. Speech-to-text was performed, and regular text search was augmented with phonetic “sounds-like” search. A speaker identification algorithm was also used, and text was colored according to speaker.

Vemuri, S., C. Schmandt, W. Bender, S. Tellex, and B. Lassey. “An Audio-Based Personal Memory Aid.” In
Proceedings of Ubicomp 2004: Ubiquitous Computing,
Nottingham, UK, September 7-10, 2004, 400-17.

Supermemo:

Supermemo Web site.
http://www.supermemo.com

Wolf, Gary. 2008. “Want to Remember Everything You’ll Ever Learn? Surrender to This Algorithm.”
Wired
(April 21).

Rank Xerox EuroParc had a number of projects involving capture in aid of human memory.
Pepys
was a diary automatically created and mailed to the user each day. It was based on location information derived from Active Badges worn by the users. Episodes were recognized (e.g., “Meeting with Joe” or “Working in office”), false positives were rejected (e.g., people sharing an office should not be presumed to be constantly in a meeting), and the level of detail was reduced to a workable summary. A “video diary” (actually a series of images captured at a rate of around ten frames per minute) was also captured by cameras in the building.
Marcel
was a paper document tracking system that used a video camera mounted over the user’s desk. Documents on the desk were compared with the database of documents to identify them.
Forget-me-not
was a memory aid system. It logged e-mail, file sharing, printing, and telephone calls and supported browsing of a user’s diary, with events filtered by when, where, or who to aid recall. A study compared three conditions: (1) no computer support, (2)
Pepys,
(3) “video diary.” They found
Pepys
improved recall, and video did even more. People and objects were the most common memory cues.

Eldridge, Margery, Michael Lamming, and Mike Flynn. 1992. “Does a Video Diary Help Recall?” In A. Monk, D. Diaper, and M. D. Harri son (eds.),
People and Computers VII.
Cambridge: Cambridge University Press, pp. 257-69, also published as Technical Report EPC-1991-124, Rank Xerox Research Center, Cambridge, UK, 1992.

Lamming, M., and M. Flynn. “Forget-me-not: Intimate Computing in Support of Human Memory,”
Proceedings of FRIEND21,’94 International Symposium on Next Generation Human Interface,
Meguro Gajoen, Japan, 1994.

Lamming, M., P. Brown, K. Cater, M. Eldridge, M. Flynn, G. Louie, P. Robinson, and A. Sellen. 1994. “The Design of a Human Memory Prosthesis.”
The Computer Journal
37, no, 3:153-63.

Lamming, M. G. “Using Automatically Generated Descriptions of Human Activity to Index Multi-Media Data.” IEEE Colloquium on Multimedia Communications and Applications, February 7, 1991, pp. 5/1-5/3.

Lamming, M. G., and W. M. Newman. 1992: “Activity-based Information Retrieval: Technology in Support of Personal Memory.”
Personal Computers and Intelligent Systems: Information Processing ’92.
Amsterdam: North-Holland, 68-81.

Newman,W. M., M. A. Eldridge, and M. G. Lamming. 1991. “Pepys: Generating Autobiographies by Automatic Tracking.”
Proceedings of the Second European Conference on Computer-Supported Cooperative Work—ECSCW ’91.
Dordrecht, Netherlands: Kluwer, 175-88.

The
SPECTOR
project looks at how to use
Pepys
-like automatic diaries to develop a model of the user and perform machine learning to help the user.

Kröner, Alexander, Stephan Baldes, Anthony Jameson, and Mathias Bauer. “Using an Extended Episodic Memory Within a Mobile Companion.” Pervasive 2004 Workshop on Memory and Sharing of Experiences, Vienna, Austria, April 20, 2004.

The
Infinite Memory Multifunction Machine (IM
³
)
was a system to automatically capture every document that a user copies, faxes, or prints. It was able to automatically detect duplicates, and was quite successful in automatically filing new documents into a user’s existing file hierarchy based on word counting and image analysis. A study with twenty users over two years showed that the average age of a retrieved document was forty-four days, with 10 percent of all accesses being for documents older than six months. This debunked a common conjecture that old documents would virtually never be needed.

Hull, Jonathan J., and Peter Hart. “The Infinite Memory Multifunction Machine (IM3).”
Pre-Proceedings of the Third IAPR Workshop on Document Analysis Systems,
Nagano, Japan, November 4-6, 1998, 49-58.

Hull, Jonathan J., Dar-Shyang Lee, John Cullen, and Peter Hart. “Document Analysis Techniques for the Infinite Memory Multifunction Machine.”
Proceedings of the 10th International Workshop on Database and Expert Systems Applications,
Florence, Italy, September 1-3, 1999, 561-65.

Hull, Jonathan J., and Peter E. Hart. 2001. “Toward Zero Effort Personal Document Management.”
IEEE Computer
34, no. 3 (March).

Here are several works that discuss expanding our definition of mind to encompass e-memories. For instance, David Chalmers, who says in an interview with
The Philosophers’ Magazine,

When bits of the environment are hooked up to your cognitive system in the right way, they are, in effect, part of the mind, part of the cognitive system. So, say I’m rearranging Scrabble tiles on a rack. This is very close to being analogous to the situation when I’m doing an anagram in my head. In one case the representations are out in the world, in the other case they’re in here. We say doing an anagram on a rack ought be regarded as a cognitive process, a process of the mind, even though it’s out there in the world.

. . . A whole lot of my cognitive activities and my brain functions have now been uploaded into my iPhone. It stores a whole lot of my beliefs, phone numbers, addresses, whatever. It acts as my memory for these things. It’s always there when I need it.

Baggini, Julian. “A Piece of iMe: An Interview with David Chalmers.” 2008.
The Philosophers’ Magazine,
Issue 43 (4th Quarter).

Clark, Andy. 2008.
Supersizing the Mind.
Introduction by David J. Chalmers. New York: Oxford University Press.

Noë, Alva. 2009.
Out of Our Heads: Why You Are Not Your Brain, and Other Lessons from the Biology of Consciousness.
New York: Hill and Wang.

Jill Price has an astonishing memory, and it seems to be a burden to her. The human mind seems to improve memory at the price of unwanted recall—this won’t be the case for e-memory.

Price, Jill, and Bart Davis. 2008.
The Woman Who Can’t Forget: The Extraordinary Story of Living with the Most Remarkable Memory Known to Science.
New York: The Free Press.

Stephen Wiltshire uses a photographic memory to amaze people with his art.

Adams, Stephen. 2008. “Stephen Wiltshire, the Human Camera Who Drew London from Memory.”
Telegraph
(April 3).

The Stephen Wiltshire Gallery Web site.
http://www.stephenwiltshire.co.uk

4. WORK

Regarding DARPA’s LifeLog and ASSIST:

Archived copy of the LifeLog Proposer Information Pamphlet,
http://web.archive.org/web/20030603173339/http%3a//www.darpa.mil/ipto/Solicitations/PIP_03-30.html

Safire, William. 2003. “Dear Darpa Diary.”
The New York Times
(June 5).

Shachtman, Noah. 2004. “Pentagon Kills LifeLog Project.”
Wired
(February 4).

———. “Pentagon Revives Memory Project.”
Wired
(September 13).

The soldier patrol story is based on Thad Starner’s talk at CARPE 2006, but also borrow’s from other publications related to ASSIST. Special thanks to Daniel Ashbrook.

Minnen, David, Tracy Westeyn, Peter Presti, Daniel Ashbrook, and Thad Starner. “Recognizing soldier activities in the field.”
Proceedings of International IEEE Workshop on Wearable and Implantable Body Sensor Networks (BSN)
, Aachen, Germany, March 2007.

Schlenoff, Craig, et al. “Overview of the First Advanced Technology Evaluations for ASSIST.”
Proceedings of Performance Metrics for Intelligent Systems (PerMIS) 2006
, IEEE Press, Gaithersburg, Maryland, August 2006.

Stevers, Michelle Potts. “Utility Assessments of Soldier-Worn Sensor Systems for ASSIST.”
Proceedings of the Performance Metrics for Intelligent Systems Workshop
, 2006.

Starner, Thad. “The Virtual Patrol: Capturing and Accessing Information for the Soldier in the Field.”
Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences,
Santa Barbara, California, 2006.

Glass Box:

Cowley, Paula, Jereme Haack, Rik Littlefield, and Ernest Hampson. “Glass Box: Capturing, Archiving, and Retrieving Workstation Activities.”
Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences,
Santa Barbara, California, 2006.

The Microsoft Research VIBE team, led by Mary Czerwinski, has developed a number of excellent visualizations, including time spent on your computer, and browsing your e-memories.

VIBE Web page.
http://research.microsoft.com/en-us/groups/vibe

Brush, A. J., Brian Meyers, Desney Tan, and Mary Czerwinski. “Understanding Memory Triggers for Task Tracking.” In
Extended Abstracts at CHI 2007 Conference on Human Factors in Computing Systems,
Association for Computing Machinery, Inc., April 2007.

Smith, Greg, Mary Czerwinski, Brian Meyers, Daniel Robbins, George Rob ertson, and Desney Tan. 2006. “FacetMap: A Scalable Search and Browse Visualization.” In
IEEE Transactions on Visualization and Computer Graphics
.

The wearable
Remembrance Agent
is hardware consisting of a one-handed chording keyboard with a heads-up display, along with radio and infrared receivers. Location is detected using radio location beacons and people are detected using infrared name badges. It runs note-taking software that selects old notes to show you based on your current location, people it detects around you, and the text of notes that you are presently writing. A desktop version of the
Remembrance Agent
operates within a text editor and brings up relevant items in a window based on what you are typing. A contextual retrieval application called
Margin Notes
has also been developed for Web browsing.

Rhodes, Bradley. 2003. “Physical Context for Just-in-Time Information Retrieval.”
IEEE Transactions on Computers
52, no. 8 (August): 1011-14.

———. 1997. “The Wearable Remembrance Agent: A System for Augmented Memory.” Special Issue on Wearable Computing,
Personal Technologies Journal
1:218-24.

Rhodes, Bradley J. “Margin Notes: Building a Contextually Aware Associative Memory” (html), to appear in
The Proceedings of the International Conference on Intelligent User Interfaces
(IUI ’00), New Orleans, Louisiana, January 9-12, 2000.

Rhodes, Bradley, and Pattie Maes. 2000. “Just-in-Time Information Retrieval Agents.” Special issue on the MIT Media Laboratory,
IBM Systems Journal
39, nos. 3 and 4: 685-704.

Rhodes, Bradley, and Thad Starner. “The Remembrance Agent: A Continuously Running Automated Information Retrieval System.
The Proceedings of the First International Conference on the Practical Application of Intelligent Agents and Multi Agent Technology
(PAAM ’96), London, UK, April 1996, 487-95.