X-Ray Vision: No Longer Comic Book Hokum

By John Oncea, Editor

Bringing a new sensing modality to AR systems through networked RF sensing, MIT researchers have developed a headset that gives users the ability to perceive what used to be invisible to the human eye.

I’m not a comic book collector but, as a kid, I owned my fair share of them. In fact, one of the best things about visiting my grandparents was leafing through the collection of comic books my uncle had left behind when he moved out. Beetle Bailey, Little Lulu, Nancy and Sluggo … they were great. *

But the reason I bring this up is because of the ads in the back of the comics. Magic cards, sea monkeys, joy buzzers, smoke bombs, and, of course, X-ray glasses. For only $1 (plus 25 cents for postage and handling) you could surprise your friends with amazing illusory X-ray sight. See through fingers – through skin – see yolk of egg – see lead in a pencil … the options were limitless.

No. Wait. Turns out, there were limits.

Now, some 50 years later, we still don’t have X-ray glasses, but we have the next best thing: augmented reality with non-line-of-site perception.

* Well, maybe not great. But whatever they were it’s a fantastic memory.

X-AR From MIT

Let’s define terms so that we’re all on the same page using Techopedia as our source.

• Augmented reality: Augmented reality (AR) is a type of interactive, reality-based display environment that takes the capabilities of computer-generated displays, sounds, text, and effects to enhance the user’s real-world experience.
• Non-line-of-site perception: Non-line of sight is a term often used when the radio transmitter and receiver are not in the direct visual line of sight, and this is dealt with by the use of multiple paths in signal propagation. Non-line of sight can be overcome with the use of antennas and other such communication devices.

Put these two technologies together, as researchers at the Massachusetts Institute of Technology (go, Engineers!) did, and you have a little something they’re calling X-AR. This new technology has many applications in retail, warehousing, manufacturing, smart homes, and more and can help users find missing items and guide them toward these items for retrieval.

“Our whole goal with this project was to build an augmented reality system that allows you to see things that are invisible — things that are in boxes or around corners — and in doing so, it can guide you toward them and truly allow you to see the physical world in ways that were not possible before,” said Fadel Adib, who is an associate professor in the Department of Electrical Engineering and Computer Science, the director of the Signal Kinetics group in the Media Lab, and the senior author of a paper on X-AR.

The Science Behind The Magic

According to MIT Media Lab, “X-AR uses wireless signals and computer vision to enable users to perceive things that are invisible to the human eye (i.e., to deliver non-line-of-sight perception). It combines new antenna designs, wireless signal processing algorithms, and AI-based fusion of different sensors.” Three innovations are at play here:

• AR-conformal wide-band antenna that tightly matches the shape of the AR headset visor and provides the headset with Radio Frequency (RF) sensing capabilities. The antenna is flexible, lightweight, and fits on existing headsets without obstructing any of their cameras or the user's field of view.
• AR-Based SAR localization algorithm that fuses RF sensing from the antenna and visual sensing from the headset's cameras to localize RF-tagged objects, even when they are hidden from the view, as the user naturally walks in the environment.
• RF-Visual Verification Primitive that fuses RF and vision to deliver actionable tasks to end users such as picking verification.

Co-author of the X-ray vision paper, Aline Eid, now an assistant professor at the University of Michigan (go, Wolverines!), said the biggest challenge was designing an antenna that would fit on the headset. “It needed to work without obscuring the cameras or obstructing its operations.” Adib added, “This matters a lot since we need to use all the specs on the visor.”

The researchers used a simple, lightweight loop antenna and gradually altered its width, and added gaps to boost bandwidth until they created the perfect X-ray vision device. Additionally, the antenna was optimized to send and receive signals when attached to the visor.

“Once the team had built an effective antenna, they focused on using it to localize RFID-tagged items by leveraging a technique known as synthetic aperture radar (SAR),” writes Interesting Engineering. “SAR allows X-AR to utilize visual data from the headset’s self-tracking capability to build a map of the environment and determine its location within that space.”

“While it presented a challenge when we were designing the system, we found in our experiments that it works well with natural human motion. Because humans move around a lot, it allows us to take measurements from lots of different locations and accurately localize an item,” said co-author Laura Dodds.

MIT reports X-AR can locate hidden items with an accuracy of less than a foot (9.8 centimeters) and it achieves 96% accuracy in picking verification. “These results show that X-AR is successful in extending AR systems to non-line-of-sight perception, with important implications to warehousing, retail, e-commerce fulfillment, and manufacturing applications.”

However, as Giant Freakin Robot ** points out, “The gadget doesn’t work exactly like Superman’s Kryptonian powers.” X-AR uses radio signals to locate items that are labeled with radio frequency identification (RFID) tags. “These objects appear as a hologram which allows users to locate the items faster. Although RFID systems use several antennas located meters apart, researchers only needed one that could achieve a bandwidth strong enough to alert the tags.”

** C’mon. Isn’t that one of the best website names you’ve ever come across?

Not Quite Perfect But They’re Getting There

As it currently stands, X-AR – while unique – isn’t ready for prime time. But now that MIT has demonstrated its success, they plan to explore how different sensing modalities like WiFi, mmWave technology, or terahertz waves could be used to enhance its visualization and interaction capabilities. They could also enhance the antenna so its range can go beyond 3 meters and extend the system for use by multiple, coordinated headsets.

“Because there isn’t anything like this today, we had to figure out how to build a completely new type of system from beginning to end,” says Adib. “In reality, what we’ve come up with is a framework. There are many technical contributions, but it is also a blueprint for how you would design an AR headset with X-ray vision in the future.”

This research was supported in part by the National Science Foundation, the Sloan Foundation, and the MIT Media Lab and the paper can be found here.