#New augmented reality applications assist astronaut repairs to space station

Most often, communications delays between the International Space Station crew and ground are nearly unnoticeable as they are routed from one Tracking and Data Relay Satellite to another as the station orbits about 250 miles above Earth. As NASA prepares to explore the Moon, about 240,000 miles away, and eventually Mars, which averages about 245 million miles away, NASA is developing tools to increase astronaut autonomy to operate spacecraft or systems without assistance from the Mission Control Center at NASA’s Johnson Space Center in Houston because communication delays from Earth will last longer.

The T2 Augmented Reality (T2AR) project demonstrates how station crew members can inspect and maintain scientific and exercise equipment critical to maintaining crew health and achieving research goals without assistance from ground teams.

To kick off the T2AR activities in orbit in April, astronaut Soichi Noguchi of the Japan Aerospace Exploration Agency (JAXA) was tasked with maintenance for one of the space station crew’s pieces of exercise equipment, the T2 Treadmill. The inspection procedure is typically available as a PDF document to be accessed on a computer or tablet, which can be hard to hold while also operating tools or flashlights or examining equipment in a tight space. This time, no extra handheld instructions or communication with ground teams in Mission Control were necessary since the information was all in plain sight. Using the Microsoft HoloLens augmented reality (AR) goggles and armed with novel procedure tracking software NASA developed, Noguchi had step-by-step guidance and cues to assist in the work without referring to a separate screen.

T2AR is the first in-space operational use of the HoloLens in combination with custom-built AR software, which enables an astronaut to perform unassisted maintenance and inspections on a major piece of crew support hardware. This investigation builds on the Sidekick experiment former NASA astronaut Scott Kelly conducted in 2016.

This novel demonstration used 3-D directional cues to direct the astronaut’s gaze to the proper work sites and displayed the procedure instructions. The device followed an astronaut’s verbal instructions to navigate procedures and displayed AR cues and procedure text over the hardware as appropriate for the procedure step being performed. The system also provided supplemental information, such as instructional videos and system overlays, to assist in performing the procedure.

“AR tools hold the promise of allowing us to pre-package guidance and expertise,” says International Space Station associate scientist Bryan Dansberry at Johnson. “The space station is the perfect platform to test out AR systems and refine these tools so they will be ready when future astronauts need them. Closer to home, these tests help to mature software and AR technology now so expertise and support are available in remote locations around the world.”

Since that first activity with Noguchi, astronaut Thomas Pesquet of ESA (European Space Agency) and NASA astronaut Megan McArthur have also used the AR application aboard the space station. The tests provided researchers with information about how the technology does and does not work to assist crew inspection maintenance procedures. With the completion of this maintenance activity, nine more test sessions remain in the technology demonstration plan.

While this demonstration was currently limited to the critical T2 Treadmill, the platform is designed to be used across a wide variety of space station needs in the future, leading to increased crew efficiency and activity execution accuracy. In addition to use on the space station, this technology could aid astronauts on future journeys to the Moon and Mars, when communication delays are greater, by allowing them to conduct tasks without waiting for further direction from Earth.

NASA’s Advanced Exploration Systems division sponsors this technology demonstration aboard the microgravity laboratory to advance future human and robotic exploration missions as part of NASA’s Moon and Mars exploration approach.