“Consecration” is something you hardly see anymore: a Catholic horror movie that isn’t about exorcism. Yet after decades of “Exorcist” knockoffs, moviegoers have been conditioned to anticipate the clichés of demonic possession. We expect them to be delivered, and in a certain way they always are. For moviemakers have been conditioned that way too.

Set in a remote convent in the Scottish highlands, “Consecration” presents the audience with a sinister Mother Superior (Janet Suzman) who talks about God as if he were the devil, along with a batch of young nuns who smile with too much cultish devotion (one wears an eyepatch because she dug her own eye out after she thought she saw Satan). We keep waiting for the other demonic shoe to drop — to see a nun possessed, or a group of them gathered in a secret ceremony to conjure the Beelzebub of their dreams. All of that does sort of happen, though not with the standard terror-trope formality we expect. And there isn’t an exorcist in sight.

There is only Grace (Jena Malone), a British ophthalmologist who arrives to investigate the violent death of her brother, Michael (Steffan Cynnydd). Did he murder Father Carroll in the chapel and then take his own life by plunging off a wuthering-heights cliff? That’s the official story. But Grace, a humanist skeptic who looks, in the early scenes, like a rather cautious denizen of Swinging London, doesn’t trust anything about religion. She’s more than willing to pin the crime on demons of the human kind.

For a while, so are we, as “Consecration” takes the form of a dark-side-of-Sunday-school murder mystery. There’s a Scottish cop hovering around the edges. And there’s Father Romero, an emissary from the Vatican, played by Danny Huston with his usual insidious mixture of bonhomie and threat. The set-up seems intriguing, especially when Grace greets the Mother Superior’s dire theology by dismissing it to her face as “bullshit.” That’s a tense encounter, though you can’t help but notice that the rest of the film has that gloomy, muddled indie-horror-film rhythm — a desultory stasis passing itself off as atmosphere. You might go in expecting an “elevated” IFC Midnight version of “The Exorcist,” but “Consecration” is closer to “Black Narcissus” directed by Roger Corman.

British actors play Americans in movies every day. It doesn’t happen as often the other way around, but in “Consecration” Jena Malone doesn’t just sport a casually impeccable British accent. She becomes British — her mood and manners, the way she rocks the sweaters and bangs and debonair politeness. She creates a compelling character, only to see the film’s director, Christopher Smith, swallow her up in all the ecclesiastical gothic malarkey.

The movie is about the mystery of Michael’s suicide-that-probably-wasn’t. It’s about Grace’s luridly extreme family past (which is insane enough to feel like a separate movie), and the visions she has going back to medieval times. And it’s about the identity she assumes, or comes to realize is really her. She’s like the hero of Ari Aster’s “Hereditary,” ushering the audience into an otherworldly existence from the inside. (She’s also like Regan in “The Exorcist” if Regan were her own exorcist.) Through it all, however, “Consecration” grows increasingly murky and blood-drenched and contrived. The scene where Grace goes to visit her monstrous father (Ian Pirie) in prison nudges the movie off the rails. And after a fair amount of jabber about a powerful relic, we learn what — or who — the relic is. It’s supposed to be a real whoa, but at that point you may just feel like casting out the demon of sketchy pretentious sensationalist filmmaking.