The title of this brutal, violent and very macho prison movie from director David Mackenzie means "transferred prematurely from juvenile detention to adult jail". The film's press pack came with a glossary explaining to reviewers some of the other code-words: "kanga" meaning officer; "tech" meaning mobile phone; "kick off back door" meaning anal sex, and "straightener", meaning pre-planned fight. For me, this last word has a certain kind of sad irony and prose-poetry. It is the debut screenplay from Jonathan Asser, a psychotherapist who has experience treating long-term prisoners with anger-management issues. That phrase itself is perhaps a kind of official code, which looks increasingly euphemistic as the film progresses.

Jack O'Connell is Eric, a 19-year-old who has been starred up, upgraded to adult prison two years early because he is just too violent to be contained at the juvenile level, and as the phrase implies, it confers a kind of twisted celebrity status on the scary new tough guy. But on arriving at the institution (in time-honoured prison-drama style, the film begins with the clangs and buzzes of secure doors opening and closing under the credits) Eric discovers the situation is more complicated than he or anyone could have expected. His father, Nev, played by Ben Mendelsohn, is at the prison too – the man whose neglect contributed so much to Eric's perennially boiling rage. Nev is high up in the pecking order of menace and threat, and an important part of the gangster power structure to which a weak and corrupt management has effectively devolved control of the prison. Nev's confused and clumsy attempts to protect Eric, and somehow reclaim some parental respect, throw petrol on the flames, especially when he intervenes in the group run by the idealistic therapist, Oliver, played by Rupert Friend.

Ugliness, shame, depression and fear are all pungently present, especially when Eric feels the only way he can assert himself on arriving is with a display of reckless violence: a kind of sub-Hannibal Lecter refusal to be cowed by any number of guards. Prison is how society deters violence: the prospect of going to horrible prisons like this. But for people already in prison? And with little or no hope of  getting out in the next couple of decades? The question of how to deter their violence while inside is harder to answer.

Asser is someone who has dealt with prisoners; he knows whereof he speaks, and the parts of the film that feel really relaxed, and have the tang of authenticity and direct experience, are the group-therapy scenes; the scenes in which tough prisoners sitting uneasily in a circle have to learn to talk about their emotions. As for the rest of the film, I wondered if the drama had grown out of the ripe anecdotes that the prisoners had been telling their therapist. It seems as if the fictional therapist himself has anger and aggression problems, but it isn't clear if this means that Oliver has somehow been poisoned by the prison's queasy atmosphere, or gone native in some way, or if we are supposed to see this as part of the heroism that gained him respect among the prisoner alpha-males.

The father-son relationship is excruciatingly embarrassing for both Eric and Nev in jail, and in fact, it seems as if embarrassment is the only emotion available for either man – an emotion that is inevitably transformed into violence. O'Connell and Mendelsohn give robust and committed performances, although their relationship is opaque. It is hard to forget the relationship of Daniel Day-Lewis and Pete Postlethwaite in Jim Sheridan's In the Name of the Father (1993), as the wrongly accused Gerry Conlon, discovering to his horror that his father Giuseppe is in prison with him – an unthinkable transgression and humiliation. And, of course, there is Ray Winstone's queasy demand in Alan Clarke's Scum (1979): "Who's the daddy?" The idea of paternal authority gets disfigured into violence, and often into sexual assault, although this film pretty much steers clear of that much whispered-about feature of prison life. One contributor to Oliver's therapy group insists that it is characteristic of American prisons, not British ones. Maybe.

Finally, Starred Up weakens and becomes a little sentimental, and there is something cliched about the way it tries to resolve the drama. For such a violent movie, it pulls its punches a little at the last. But not before it has given us a bleak picture of the defeated misery of a certain kind of violent man, locked in a prison of his own making.