In an ideal world, Simon Pegg would physically assault his audience. "People need to be poked in the face," he announces, gripped suddenly by a passion so intense it causes him to surface from the fog of jetlag and shove aside his walnut and avocado salad. (He only recently returned to the UK from shooting Star Trek 2 in Los Angeles, and admits to needing help with key nouns and adjectives.) "Maybe not a poke in the face," he continues after a second's thought. "But the ribs, at least. I like the idea of confounding audiences to a degree, challenging their expectations. We are given what we expect so much now. There's this desperate fear of upsetting anyone. All we get in the cinema are 3D fireworks displays. But interaction is more important than passive watching; that's just a waste of the art form. My attitude goes back to Howard Barker's book Arguments for a Theatre, and his insistence that it should be painful and awkward and difficult for the audience."

Painful and awkward and difficult: not words that the distributor of Pegg's new film will be in any hurry to splash over the poster. But it is this movie, the horror-comedy A Fantastic Fear of Everything, that has led us on to the subject of wrongfooting the average punter. Pegg plays Jack, a neurotic writer whose heebie-jeebies get the better of him after he researches a book about serial killers. This is Pegg's 1408, his Cast Away: for the majority of the running time, he is on screen alone, at least until Jack is driven to the brink of a breakdown by visiting the one place that is the nexus for all his fears: the local launderette.

I admit to Pegg that silent bafflement was the predominant reaction at the screening I attended, which is what has led us on to a discussion of audience expectations. And who better to help us out on the subject than Crispian Mills, for whom A Fantastic Fear of Everything marks his film-making debut. "I've watched it with audiences," says the 39-year-old Mills, who co-directed the movie with the pop-promo wizard Chris Hopewell. "There is definitely that moment of bewilderment in the room when the audience realises that the main action really is going to be played out entirely in the launderette. A friend of mine said at one of the test screenings that the film is like shock therapy: you can't anticipate how people are going to react."

Mills has joined Pegg and myself in the quiet corner of a Soho members' club; he is slightly late and wearing a faded T-shirt, as is the prerogative of any rock star, current or former. Let's get the formalities out of the way: yes, he is that Crispian Mills, frontman and linchpin of the psychedelic Britpop outfit Kula Shaker, best known for hits such as Tattva, Hey Dude and Grateful When You're Dead but most definitely not Deutschland Über Alles. The Crispian-Mills-thinks-the-Nazis-were-misunderstood story, which helped kill off the band's first flourish in the late 90s, has long since been explained away as a combination of fame-induced pop-star hubris and media hysteria.

It's not such a surprise that Mills should branch into film-making given the cinematic stock from which he is descended. His grandfather was Sir John Mills, and he is the son of a genuine Brit-cinema power couple from the days before that phrase was coined – his mother is the luminous Hayley Mills, his father the late writer-producer-director Roy Boulting (of the Boulting Brothers).

Mills always said that as a child he assumed he would end up making films. What took him so long? "When I was younger, I didn't give it too much thought. I just knew that was what everyone else did in my house, and they all got to show off about it at Christmas when they were playing charades, so I would do that eventually too. But I fell in love with music and became completely immersed in that world. It wasn't until I was 27 or 28 that I realised how obsessed I was with film."

The catalyst was twofold. First, Kula Shaker disbanded after a brace of albums (though they recorded again after reforming in 2004). Then Mills read Easy Riders, Raging Bulls, Peter Biskind's celebration of the excesses of the American new wave. "I felt such a connection with that world," he says. "I understood what that struggle was like, having heard so much about it from my family. Even someone like Dickie Attenborough, whom I'd grown up around – you think of him as the establishment, but he's fiercely independent. People like him and Bryan Forbes were very much masters of their own destiny. Those fights continue."

His parents met on The Family Way, but it is their collaboration on the sinewy thriller Twisted Nerve that is perhaps more pertinent to Mills's directorial debut; even as it is prodding us toward laughter, A Fantastic Fear still tries to make our skin crawl. "Horror and comedy overlap so often because they both elicit an immediate emotional response," says Pegg, who has form in this hybrid genre, with Shaun of the Dead and Steve Coogan's mock-Hammer series Dr Terrible's House of Horrible on his CV. "Ghost stories and jokes have the same structure: setup and pay-off. Except that one provokes fear, the other laughter. Laughter is a fear response anyway; it's about trying to make sense of irregularities in our reality. It's what makes horror and comedy such obvious bedfellows."

In its structure, which has Jack experiencing psychological breakthroughs that help him to face his demons, the movie sometimes resembles a psychoanalytic teaching aid. It's clearly a film made by someone who has undergone a lot of therapy, isn't it? "Well, they do say you've got to write about what you know," laughs Mills.

By his own account, he rather lost his way in the rush of fame and approbation that dominated his early 20s. "The younger you are, the more difficult it is not to let it get to you," he says sagely. "Even if you know it's all bullshit. You become the centre of other people's universes as well as your own, which isn't healthy. When you reach America, English bands in particular take the piss – you're just treated so well, you can't believe it. And it does get to you." He pauses and looks me hard in the eye. "It really does get to you."

Pegg has seen something comparable happen in the film industry. "It probably occurs with actors a little less. It's something that is expected of musicians, you know, the TV out of the window. With actors it manifests itself as disappearing into the trailer for hours at a time. I'm 42 now, I'm not some youngster believing my own hype, and I always enjoy working on a set where everyone's happy. I couldn't walk on set two hours late having refused to come out of my trailer, and feel the grips looking at me thinking: 'Who's that arsehole?' I'm not OK with that, but I can see how it happens." Mills is nodding emphatically. "You become a cunt. And if you're lucky, the other guys in the band will look out for you and tell you that you're being a total cunt."

Next up for Pegg is the sci-fi comedy The World's End – his long-awaited third movie with the writer-director Edgar Wright, with whom he has done his fizziest, funniest work, from the Channel 4 series Spaced through to Shaun of the Dead and Hot Fuzz. They're currently corresponding with the BBFC over how many times they can use the c-word in The World's End without forfeiting a 15 certificate. "The BBFC wrote us a lovely letter back," says Pegg cheerfully, "and let us know we could say it once, aggressively, toward a woman, but five times in casual conversation."

Mills, on the other hand, doesn't seem ready yet to move on from A Fantastic Fear, and what it means to be releasing it at last into the world. "It's only now I realise what a massive invasion of privacy it is to put out something so deeply personal. This is a story about a guy who hates himself, after all. The way Simon plays it is very funny, but it's all there: the self-loathing of the drama queen."

A Fantastic Fear of Everything is on general release.