Following the success of 2009’s groundbreaking Avatar, James Cameron began writing a new story that would take Jake Sully (Sam Worthington), Neytiri (Zoe Saldaña) and their children on an emotional journey in a new region of Pandora — moving from its bioluminescent forest to its seas in the hopes of protecting their family from the Sky People. “Our family is our fortress,” Jake Sully says in Avatar: The Way of Water, which this past week won a BAFTA for its visual effects and swept the feature competition at the Visual Effects Society Awards.

The director had been working with Weta FX and its senior VFX supervisor Joe Letteri, winner of four competitive Oscars, since they first teamed on Avatar. Letteri emphasizes that everything they do from a VFX standpoint is about supporting the story, but when he saw the new script, he knew they had their work cut out for them. “We started to put a lot of effort into thinking about all aspects of what that would mean to create characters in water,” he says.

For the uninitiated, water simulation and emotive digital characters are each sizable VFX challenges — among many that the filmmakers would address in making this movie — and they’re part of an overall workflow that involved collaboration among all departments. Notes Letteri, “One of the first things that came up was, ‘If you’re going to get [the actors’] motion right, you have to capture the performers in water.’ “

Explains Cameron: “No one had ever done performance-capture underwater. There were people that told us it wasn’t even possible — and certainly not possible at the kind of scale that we needed.” But in the end, he reports, the filmmakers would develop a tank possessing the capabilities needed for filming, including “generating waves and underwater currents and bottom surge and shore-break surf; we had a movable platform system that could create beaches and shorelines, modular set construction and all that.” And the team had to develop the methodology for the underwater performance-capture. Cameron adds: “I knew it was possible, I just didn’t know exactly how it would wind up being done. It took us about a year and a half to figure that out.”

Key to the film’s success was training the actors, who would have to perform underwater wearing suits with tracking markers. “None of them were really divers [when they started the project],” the director says, noting that it was vital to make the system user-friendly for the actors as well as the camera crew. They also needed to address story points such as riding the skimwings — creatures that can both swim underwater and hover above the surface in the air. “We went out to the ocean first, and we had mock-ups of our creatures that could actually move around very rapidly underwater,” Cameron says.

Senior animation supervisor Dan Barrett says the skimwings were inspired by flying fish that “can sustain flight for quite a while. They can do as the skimwings do; they can fully leave the water or they can have this kind of wing and ground effect where they’re receiving lift off the water surface to their wings and propelling themselves, thrust from the tail in the water.”

To get the vital facial performances, Weta FX developed a new stereo camera head rig, and accompanying software for facial capture, to bring the actors’ undiluted performances to their characters. Explains Letteri: “[The software] is a neural network that we built as an animation tool that lets us try to figure out how the muscles react to each other. It gives the animators a lot more control in interpreting a performance than we’ve ever had before.”

Emphasizes Cameron: “It’s 100 percent actor-driven. Some­times our biggest challenge can be in a simple two-character dialogue scene. You have to have every nuance of what the actor does. As a director and working with the actors, it’s a very pure art form. It’s like a stage rehearsal.”

Jake and Neytiri’s adopted teenage daughter, Kiri, played by Sigourney Weaver, involved some delicate consideration. In the story, Kiri is the biological daughter of Grace, the scientist played by Weaver in 2009’s Avatar. “What you’re seeing onscreen is Sigourney’s performance, 100 percent,” says Barrett. “All of the underwater stuff, she did it. She could sit at the bottom of that pool for minutes, doing these performances. It was really quite impressive.”

To get the character just right, the crew at Weta assembled photos of Weaver as a teenager, as well as footage from some of her early movies, to give the team nuanced references in how her face moved at that age because, as all faces do, “the face changes a fair bit the older we get, muscles change,” Barrett says. To ensure that all motion was exact, there were “certain occasions” when a body or stunt double was involved for specific types of movement.

In a movie that is set so heavily in the sea, you can’t uncouple the characters from the hurdles of water simulation. “We’ve all seen water rendered before, [but in this film] we were going to have a lot of character interaction with water,” says Letteri. “Jim was going to be in this virtual ocean with a virtual camera and a virtual splash housing as characters are having dialogue and action.”

The digital water in the film was created by artists using Weta’s latest Loki simulation software after — pardon the pun — the Weta FX team did a deep dive into understanding water. “What’s the color of the water? How should the light actually travel through it?” were some of the questions they addressed, says Metkayina Village and reefs VFX supervisor Pavani Boddapati. Adds Jonathan Nixon, effects supervisor for water and fire, “A good portion of this film is computer-generated water because we just got to a point where it looked so good, it was easier just to replace [the actual water] and integrate it [into the shots].”

As with the 2009 movie, when filming on set, Cameron shot native 3D; Letteri notes that they also innovated on these sets. “Basically [we developed] live depth compositing,” he explains. “What that meant is that when Jim was walking through the set and a character was interacting with a virtual character, that virtual character could appear in the live feed, but in the proper place in [3D] depth.” This was enabled by newly developed AI-driven software that effectively took the images from the camera system and calculated 3D depth on the fly.

Another new tool gave actors reference for eyelines and performance, with a cable-suspended camera system, but the team replaced the camera with a monitor. Says Letteri: “All the live-action characters, when they had to interact with CG characters, had something flying around the set with the performance in the right place in space. [It provided] the ability for the live-action characters to play live off their CG counterparts.

“These are just all tools to build this story,” he concludes. “We knew it was going to be a big story and a lot of emotional beats with the characters, and that’s really where we were putting all the effort.”

This story first appeared in a Jan. stand-alone issue of The Hollywood Reporter magazine. To receive the magazine, click here to subscribe.