“Thinking of Melville, thinking of Poe, thinking of Mark Twain and Vachel Lindsay, thinking of Jack London and Tom Wolfe, one begins to feel there is almost no way of becoming a creative writer in America without being a loser.” So declared the great Nelson Algren, gutter bard and patron saint of Chicago letters, a man who recognized the quiet dignity of loserdom and accepted more than a little of it for himself. He rose to the top of the literary demimonde during the 40s and 50s on his writing about societal cast-offs, rough-edged characters based on the people he met in dank bars, back rooms and street corners. In training his focus on those most preferred not to think about, Algren found a bruised nobility in the down-but-not-out subjects who considered him one of their own and became a national name for it. As his celebrity declined and his output waned, however, his legacy has somewhat diminished in comparison with that of peers like mutual admirer and friend Ernest Hemingway.

That’s where Michael Caplan, director of the new documentary Algren, hopes to come in. After more than a decade spent on a production interrupted by disappearing and reappearing funds, the professor/film-maker has completed his tribute to the enduring brilliance of a larger-than-life literary luminary. Caplan had a fortuitous encounter with cult-legend photographer Art Shay in 2008, who suggested that he take up Algren for a future project and offered a massive cache of up-close-and-personal photos as its basis. A completed cut screened for hometown audiences at the Chicago international film festival in 2014, only to gather dust on Caplan’s shelf for the rest of the decade, until the immobility of the pandemic lockdown compelled him to take action. He sculpted a fresh edit and with this week’s long-awaited digital release, he’ll make his bid to restore Algren’s name to the top of the writerly firmament.

“I encountered Algren first through a used copy of Man with the Golden Arm, which I read in my early 20s and I was just blown away by it,” Caplan tells the Guardian over the phone from his home in the Windy City. “I’m from Chicago, I grew up in Chicago in a very middle-class working neighborhood by the steel mills. I knew a lot of people whose families were focused on just getting by, far from the American dream. Those were the kinds of people that Algren wrote about, so his work really struck me. I always felt like he was the embodiment not just of Chicago, but of mid-20th-century America coming out of World War II, when everything was supposed to be a Ford and white picket fence. It wasn’t like that for a lot of people and that’s what Algren wrote about.”

Born to the lower half of the middle – “working-class, but not exactly poor”, Caplan explains – Algren grew up immersed in a grittier milieu than many of the well-heeled, well-educated celebrity scribes of the day. A proud product of the public school system, he planned to find work at a newspaper following his college graduation, but the Depression instead sent him on an eye-opening cross-country odyssey by freight train. During these lean days, he was pinched for purloining a typewriter from a Texas classroom and spent five rough months in the clink, during which he developed a deep identification with the assorted wretches he met there. “This introduced him to another America,” Caplan says. “That’s where the spark first happened.”

Algren got involved with the Works Progress Administration established by the New Deal to aid jobseekers, ultimately meeting Studs Terkel, Richard Wright, and “a lot of writers telling the story of people on the underside” through the arts outreach of the Federal Writers’ Project. Stimulated by their high-minded company, he began to churn out a steady stream of short stories and novels about his originally chosen cohort: alcoholics, sex workers, drug users, boxers on the take, penny-ante criminals, corrupt politicos, anyone close to the howling desperation he could sense festering beneath the wholesome exterior of the United States. He shipped out to the European front of the second world war before his career hit its peak and returned to find that “he was certainly known as a guy who could both get an award from Eleanor Roosevelt and hang out with the bums on Skid Row”, as Caplan puts it.

Though plenty of writers have fancied themselves mouthpieces of the people for taking a tour of poverty before retreating to their towers of privilege, Algren was dedicated to the squalor he stylized in his work. “To me, that’s what distinguishes him – his credibility,” Caplan says. “He really lived this life that he wrote about. He had a little apartment in Wicker Park, which was a slum mostly for Polish immigrants at the time, in almost a monastic lifestyle. He had a very stripped-down existence, and then he’d hang out at the bars where everyone knew him. This wasn’t checking a scene out for two weeks or a month. He was committed. This was his life. He hung out in high society, but to him, the real world was ‘behind the billboard’, as he said. If you didn’t know what was going on at the bottom of a society, you didn’t know what was going on, period.”

While maintaining his connection to the dregs, the success of novels like The Man with the Golden Arm and A Walk on the Wild Side catapulted Algren to the top of society’s highest echelons. His involvement with the feminist trailblazer Simone de Beauvoir and the eventual love triangle formed with Jean-Paul Sartre, fictionalized in her novel The Mandarins, helped to solidify his outsized renown. But his fortunes waned as quickly as they rose, commercial success eluding him while his contemporaries continued thriving. He went into academia as finances tightened, considered going into the flourishing Vietnam black market and dodged his taxes as anti-war protest. In 1975, a sorely needed journalism assignment profiling a wrestler led him to Paterson, New Jersey. He fell in love with its serenity and left his beloved Chicago, leaving his mantle behind. He ended his life in a semi-obscurity that’s only grown with time.

“Part of it is that America moved on,” Caplan explains. “They didn’t want to hear a story about the working class or poor. Saul Bellow was writing about middle-class people in Hyde Park and that became more desirable in that moment. Hemingway made sure that he always had a reputation as a tough guy, a guy’s guy, and Norman Mailer did the same. Even though, for a while, Algren had some of that image, he didn’t work at it so much. Once it passed, he didn’t go out and cultivate it, make sure people remembered him. He just wanted to write and thought that should be enough. Today, we know more clearly than ever that it’s not enough, people like to have an idea of the person.”

Algren belonged to a generation of hard-drinking, brawling writers that’s long since gone extinct. His bare-knuckle hyper-masculinity renders him utterly alien to a more sensitive current publishing landscape and his public feuding has migrated to other pop-cultural spheres. Caplan floats hip-hop as a present-day equivalent to the heated literary scene of Algren’s era, chuckling before conceding that “there’s something to that!” at the suggestion that he could have been the Tupac of his day. The documentary takes a stance of prudent ambivalence to his lessened stock; he was a vital talent and crucial piece of the national literary tradition, but inextricably tied to his time and place in a way that the onward march of time doesn’t necessarily favor. Caplan readily acknowledges that for all his genius, the coarseness by which Algren defined himself keeps him planted in the past. “It’s a little bit tougher when you’re open to scrutiny,” he says. “That’s something I don’t think Algren would have done well with.”

• Algren is available to rent digitally in the US on 11 January with a UK date to be announced