Exiting the New York premiere last week of his latest film, Woody Allen was delighted with the reception (and the A-list guests). But even he was aware that this level of prolific production has to end some time. "It's hard to find a story with a good beginning, middle and end that'll be entertaining to an audience," he told the Observer. "Eventually the supply will run out. I'll just keel over in my room with a stroke or something."

Allen is 78, and the reception accorded to Magic in the Moonlight, his fourth film in three years, confirmed that he is an American treasure of the first order, despite dark clouds over his private life.

The turnout of grandees at the premiere, from fashion's Anna Wintour to former New York police commissioner Ray Kelly, seemed to settle the debate over whether support for him would be withheld after allegations of child molestation were renewed in the New York Times this year. Allen is more consistently successful now than at any point in his five-decade career.

Nevertheless, it has been a rocky year. In February, Allen was publicly challenged by his adopted daughter Dylan Farrow, who reiterated claims – first levelled by Mia Farrow in the early 1990s – that Allen had sexually molested her. The claims came as Allen was about to receive a lifetime award at the Golden Globes. Allen rejected the accusation, saying that a previous police investigation had cleared him, and his statement would be his "final word on this entire matter".

Last week he dismissed a suggestion that audiences might be hit. "I don't think anyone has ever not come to a film of mine they thought they would enjoy," he told the New York Times. "Nothing keeps them away if they think they'll enjoy the film. And if they don't think they'll enjoy the film, nothing we can do ever brings them in."

He seems to have won the public relations battle, however. One veteran film agent said last week that it was significant that New York society had turned out. The original accusations had been a professional disaster for Allen, but their repetition seemed to damage ex-wife Mia Farrow – who is now perceived by some as the vengeful ex-partner – more.

Meanwhile, Allen shows no sign of reducing his workload. Magic in the Moonlight, set on the Côte d'Azur in the 1920s and starring Colin Firth as a celebrated illusionist and Emma Stone as a would-be medium, comes in the wake of Midnight in Paris and To Rome with Love. Then there was Blue Jasmine, which earned Cate Blanchett a best actress Oscar. Each film was comfortably profitable, with Midnight in Paris taking $173m at the box office.

The movie industry has ignored calls from Farrow to boycott Allen's films; he is currently at work on a new picture starring Stone and Joaquin Phoenix, and more are being readied.

While critical notices won't be published until next week, the latest film is clearly highly personal. Allen briefly performed as an illusionist. He told the Observer the film was inspired by a story he had read about the fraudulent spiritualists that the illusionist Harry Houdini was committed to exposing.

Allen is a storyteller, says David Thomson, critic and author of The New Biographical Dictionary of Film. While this may not be one of his best, "that doesn't mean that next year he won't make a very good one. Like any storyteller, some work and some don't. But why on earth do actors line up to work with him? The answer is, of course, because he's become such an institution that it is a badge of credit – almost a rite of passage – to work with him."

According to Firth, it would be mad not to exploit the years of craft contained in one directorial brain. "His writing, how he conceives characters and the plots he places them in, flatter actors," he said. Magic in the Moonlight, with Cole Porter's You Do Something to Me as its repeating soundtrack, also revisits the director's nostalgia for the elegance and aesthetics of past times.

For a director as dedicated to his work as Allen, an investigation – at 78 – into questions of spirituality and rationality is what makes the film more interesting, says Thomson: "It's a condition of creative work that creative people have great difficulties with life, so they find something that seems lifelike. He has a niche and an audience, and he makes films for a certain budget that gets enough of an audience so he can do what he wants."

That may also go to explain why Allen has never embraced Hollywood: "A natural storyteller like him could have earned a lot more money working in Hollywood, and I'm sure he enjoys it when a film does well, but I don't think that's what pushes him on." Allen, he adds, has restored himself professionally. In spirit, he's the same sceptical, neurotic character he's always been.

"I'm like Blanche DuBois," Allen said in his recent New York Times interview. "I hope in life that there's a certain amount of magic. Unfortunately, there's not enough. There are little, sporadic things one could think of as magical. But for the most part, it's grim reality."

• This article was amended on 19 July to attribute Woody Allen's quotes to the New York Times.

• Woody Allen's Magic in the Moonlight: first reviews published
• First look at Woody Allen's 1920s romance Magic in the Moonlight