PLAN 74: Article
Putting The World At Your Fingertips

SixthSense Integrates Information with Reality

The Media Lab's Fluid Interfaces Group is developing a device that can virtually turn the entire world into your personal computer. For his central role in developing it, graduate student Pranav Mistry (SM'08) has been named one of 35 top innovators worldwide under the age of 35 by Technology Review.

SixthSense brings information that is usually confined on paper or on a computer screen into the tangible world, allowing you to interact with it using natural hand gestures. When it was first introduced last year, it consisted of a web camera strapped to a bicycle helmet but the current version is more consumer friendly. When it was demonstrated at TED this year, it got a standing ovation. (See Patties Maes' presentation here.)

SixthSense consists of a camera, a mirror and a small projector, about the size of a cigarette pack, that are hung on a lanyard around your neck. The camera acts as a digital eye, seeing what you see and tracking the movements of your fingers to determine not only what you are interacting with but also how you're interacting with it. An accompanying smartphone in your pocket searches for information that is relevant to that situation, using the Internet as a data store, then aggregates the results and presents them back to you in a display.

In a bookstore, for instance, it could recognize a book you're looking at and project the text of reviews onto it from your most trusted source. The reason for your flight delay could be projected directly onto the boarding pass in your hand and a newspaper could prompt the device to search for relevant video clips and project them anywhere you like.

The drawing application lets you draw on any surface by tracking your fingertip movements. The map application lets you zoom in, zoom out or pan on a map by simply using hand gestures. You can check your email on any blank wall just by drawing an '@' in the air and check the time by simply drawing a circle on your wrist. Or take photographs by using your fingers to make a picture frame, then use any nearby surface to sort through the photos you’ve taken.

The current prototype system, built entirely from off-the-shelf components, costs about $350 but in production it wouldn’t need to cost much more than a cell phone. In this initial research phase, the team has only developed a few applications but as time goes on they envision letting others develop applications for it, as well. Future applications might include a sign language application that would 'speak out' a translation while someone was signing. Another might allow you to get a real tennis lesson on a real tennis court.

For more information on SixthSense, visit pranavmistry.com/projects/sixthsense.