“Extraordinary.”
“Pretty strange.”
“Completely weird.”
These are the professional assessments of the vision system of the seabed-dwelling mantis shrimp from Viktor Gruev, associate professor of electrical and computer engineering at the University of Illinois–Urbana-Champaign; Tom Cronin, professor of biological sciences at University of Maryland–Baltimore County; and Ilse Daly, a postdoc researcher in biological sciences at the University of Bristol in the United Kingdom.
Those scientists and other researchers are now unlocking the secrets of the mantis shrimp’s exceptional eyes, hoping the knowledge can be used to develop applications ranging from underwater cameras to medical imaging to robotics.
Around 400 species of mantis shrimp—crustaceans that typically measure four to 15 inches in length—live on the seafloor, mostly in the Indian and Pacific Oceans between Africa and Hawaii. They like to hide beneath things, sneaking out when prey comes by and whacking it with their claws, delivering a punch so strong it can shatter an aquarium window.
Mantis shrimp eyes are unlike those observed in any other animal, both mechanically and optically. They can move side-to-side and up-and-down, like the rest of us, but also torsionally, which means they can rotate their eyes in a circle on an axis. Mantis shrimp can also see a wider range of the spectrum—from ultraviolet to infrared—and in more colors than humans can. Where we see three colors (red, yellow, and blue, combining them in different proportions to see green, orange, purple, and the rest), mantis shrimp can see between 12 and 16 colors, depending on the species.
They can also detect polarized light, which is light that oscillates in only one direction, whereas the electric fields in most light arrive at a lens from multiple and often overlapping directions. This is useful because light under water is highly polarized and the polarization has multiple orientations (vertical, horizontal, and diagonal).
To see more than one degree of polarization simultaneously, mantis shrimp quickly rotate their eyes torsionally. But that would cause blurriness in most animals (or cameras), as when you move a camera to follow a fast-moving target: The background becomes less distinguishable. Yet it doesn’t seem to confuse this animal.
“It’s incredible machinery,” Gruev says.
To understand how mantis shrimp stabilize their gaze, Daly designed an experiment involving a stationary tank covered with a pattern of contrasting stripes moving horizontally, so that it looks to the shrimp like it is moving relative to the world. Eye movement in this circumstance is made to compensate for the motion. Cronin compares it to being in a slow-moving train leaving the station: “Your eyes will track backward and watch the world go by through a series of backward movements followed by quick, forward movement to replace the edge of the front of the window you’re looking through.” If you just looked straight ahead, the image would get blurry and your brain couldn’t pick up any useful information.
Mantis shrimp do backward tracking, but they also roll their eyes in a circle. “That’s counterintuitive,” Daly says. “The whole point is to stop the world from moving, so it doesn’t make sense because [torsional eye rotation] adds a new level of motion.”
So Daly took an old garbage can and painted the contrasting stripe pattern on its inside, then placed the mantis shrimp in the can. As the drum rotated, Daly figured their eyes would also roll, to stabilize the motion. But that wasn’t what happened.
“They just looked confused,” she says. Sometimes they moved their eyes in the same direction as the drum, sometimes in the opposite direction. Sometimes they just looked around. “It seems like they have absolutely no need to stabilize their eyes in that dimension.”
Even when their eyes were tilted so that what was up is now on their side, and what was on their side is now up, they made no compensation. That would make life more difficult for humans, Daly says, but “clearly they’ve developed a motion-detection system that compensates for this.” Daly’s team believes this motion-detection system is “radially symmetric, so that no matter which way their eye is in the torsional dimension, they can tell which way is up and track motion.”
This may have to do with the mantis shrimp’s polarized vision. Rapid torsional movement enables them to capture two angles of polarization more or less simultaneously. A camera would have to take two photographs at different times to capture both angles.
“This gives you worse data,” says Robert Pless, chair of computer science at George Washington University, who works on computer vision.
Gruev, who worked with Pless when they were both at Washington University in St. Louis, built a camera with the ability to see both color and polarized light simultaneously in the same set of pixels. The field of computer vision is currently grappling with questions about what sensors to deploy on driverless cars and underwater automated vehicles in various circumstances. “There might be the potential for polarization to solve some of these ambiguities, and that would be really important,” Pless says.
Pless has been working on geolocating images from the properties of their light, given the time and date the photo was taken. It turns out mantis shrimp may be doing this already. By capturing images with his camera all around the world, Gruev discovered that the polarization properties of the light would change according to where he was. Polarization with time and date revealed location. The finding will “absolutely inform my work,” Pless says.
Gruev points to other uses. “Polarization of light is an indirect measurement of the healthiness of tissue,” he says. As it deviates from its normal state to a cancerous state, the camera might be able to pick that up. His team is currently working to get the camera adopted as a point-of-care device to make it easy for doctors to use polarization information for diagnosis.
Phillip Isola, a visiting research scientist at the artificial intelligence company OpenAI, works on another area that Gruev’s camera could inform. “What’s really cool” about Gruev’s work, he says, is “the idea that there’s a lot of hidden signals in the world—something we don’t notice, like polarization patterns, can tell us more than we expect.”
One area of his work involves translating between different sensory modalities, such as a sound recording and a video of the same room, to try and find the common signal. “Having more types of sensory systems you can interrelate is a way to discover that meaningful underlying physical signal,” he says.
“When I think about biology-inspired vision systems, it’s always so much more interesting than the story you think it’s going to be when it starts,” Pless says.
This article originally appeared on Oceans Deeply, and you can find the original here. For important news about our world’s oceans, you can sign up to the Oceans Deeply email list.
