'You listen to one of those tapes and you already know how it ends," says Jane Gillooly. "The story is so predictable. But some of them really were so heartbreaking. Sometimes I got so disturbed by what I was hearing I would hide around the corner from the tape recorder."

Gillooly, 55, a director based in Boston, is explaining the peculiar process of working backwards to create her documentary Suitcase of Love and Shame. The film details the course of a long affair between a woman named Jeannie and her married lover, Tom, in the American midwest in the 1960s. Its story is told through the contents of a suitcase Gillooly bought on eBay: slides, photographs, letters and, crucially, a collection of audio tapes recorded and exchanged by the lovers themselves, which detail every aspect of their affair, from the mundane to the explicit, via mad love and despair. "There was 60 hours of material," says Gillooly. "Listening to the tapes the first time and not knowing what I would hear was where I had my most visceral responses. They felt so present. You actually felt it was happening in the room."

Gillooly's film is one of a number that builds by exploring the history of found objects. This month sees the release of Finding Vivian Maier, the story of a Chicago nanny who was also an extraordinary street photographer, but whose pictures remained unseen in her lifetime. Maier died in 2009, two years after author and historian John Maloof found a box of her negatives at auction while researching a book on Chicago history and began tracking down both the woman and her photographs.

Maier took some 100,000 pictures on her Rolleiflex camera, mostly undeveloped, and largely of everyday life in Chicago – maids on street corners, the scruffier corners of the city's Polish quarter, a young man riding on horseback through the middle of the road, but also of trips to places such as Bangkok and Italy and Egypt. They have since been exhibited around the world and published in a collection, but Maloof's film looks behind the images to try to find Maier herself, tracing the course of her life, from New York to France and back to the US, as well as finding the children she cared for.

A few years ago, Jason Bitner, one of the creators of Found magazine, also stumbled upon a stash of photographs: 18,000 formal portraits taken over the course of three decades by Indiana photographer Frank Pease at christenings, graduations, marriages and family gatherings. Bitner turned the collection into a book that would later become a documentary, revisiting some of the Pease's subjects – now adults and aged, divorced, humbled by life's turns.

Not dissimilar is Simon Aeppli's 2011 short film Come In and See the Bed, which followed the year after his father's death, in which his mother's neighbour began leaving peculiar notes in the privet hedge between their two bungalows, accusing her of conducting an affair with her middle-aged son: an unsettling stack of warnings scrawled on fragments of envelopes and slips of old paper. "I get no sleep at night time," many of them begin. "Stay in own house at night …" insist others.

For Gillooly, there was less of an element of chance in finding her subject matter. She was looking to make a film about time – inspired, she explains, by finding a collection of all her old birth control. "It was fascinating unpacking this box," she says. "They really did tell a story about sexual politics in a certain period. And I thought this would be an interesting way to build a film." One of her earlier works was a piece made for US public television in the 1990s, inspired by a photo of a dead woman, naked on a hotel-room floor, after an attempted illegal abortion in 1964."It's an horrific image that a lot of people in the US have seen. I took this very famous photograph and then I researched it and recreated the events that led up to the moment the photograph was taken."

In that film, Leona's Sister Gerri, Gillooly examines the contents of the dead woman's handbag, piecing together her story from the letters and photographs it contains to discover that the woman had sought the abortion because she was pregnant by a married man. "She was ashamed," Gillooly says. "People wanted to know why I was making a film about abortion. And I used to say to them that to me this wasn't a film about abortion, this was a film about shame."

When she came to research Suitcase of Love and Shame, she was therefore already familiar with the sexual politics of the era. "It was the same time, the same moral codes," she explains. "Birth control was still illegal [in some states] in the United States in 1964. A time when people's sexual lives were so inhibited."

Gillooly took a painstaking approach to her subject – first acquainting herself with the technology of two-track tapes before digitalising each of the recordings, and then researching who Tom and Jeannie were, the towns they lived in, the places they would meet in secret. "There was a lot of other information in the suitcase that I didn't share with the viewer, because I wanted to keep it mysterious," Gillooly says. "But I could track down addresses they were at in the early 1960s. I do know where both their houses were, so I was able to go and do some shooting in the area where they lived. I did layer in some sound from the midwest, and there's a scene in the hotel, where there's a revolving door and you hear the elevator ring … that was the actual hotel where that tape was made." It made, she recalls, for a particularly intense recording experience. "Very little had changed at the location. Even the elevator hadn't been changed. And there is a shot in the film looking out of the hotel-room window, and that was the scene they would've seen." It is not important, she feels, for the audience to know that this is the actual location. "But I do," she says. "That's important to me when I work; I need the back story in order to shape my films."

Initially, she had intended the film to be an audio piece, inspired by Derek Jarman's Blue. "My first cut didn't have any pictures at all," she says. "My first cut was going to be where you would come in, and you'd sit down and you'd listen." But gradually she began to add images – abstracts that grew into more concrete depictions but still remain open to interpretation.

"The film itself is unique in that you're staring at a screen but you're not really watching a movie," Gillooly says. "A lot of what you're seeing is in your own imagination. And you're being prompted to imagine certain kinds of things."

It was with this in mind that Gillooly chose not to include or recreate any of the more explicit scenes, judging that the audience's imagination would be more powerful. "I have the pictures they took of each other. But I think the parts that people find erotic are different depending on who the listener is, and I think that this is fascinating."

However, the first scene she shot was a group of red trees used to illustrate a scene in which Jeannie orgasms. "There's a certain kind of tree that grows here called a burning bush," Gillooly laughs. "They're gorgeous and they have that crazy reddish pink in the fall. I knew I wanted something that was just going to be really abstract. So I decided I wanted to shoot these trees, bringing them into focus when she orgasms. And it's kind of hokey and obvious in a way, but I think it worked."

The explicitness of some of the recordings is of course notable, but Gillooly says she was equally fascinated by other elements of the story – the intricate descriptions of sex, for instance, also offer insights into the era. "It wasn't so much shocking that they were talking about it," she says, "but what was surprising was that they seemed to be really uneducated about sex. Not entirely. Maybe they didn't have a word for female ejaculation then." Similarly, the behaviour of Tom's wife, diligently tracking the couple down to hotels "very really could have been because there really were these morals laws in the US, where if you couldn't prove that your husband had done something against the moral code then it would be very hard for you in court to argue in your defence: I'm sure she was very worried about being financially ruined if he left them."

The film's most affecting parts come in the couple's construction of the day-to-day intimacy an affair cannot grant: Jeannie recording the sounds of dogs barking as she showers; Tom, a doctor, reading through the entries in his diary, full of 10am meetings and afternoon appointments, delivering anecdotes about making lamp-bases, or describing every single object on his office desk. It is the sound of a man falling in love with himself, in some ways, but also supplying his lover with the sustenance to keep their affair going. "And I think that's why Jeannie was the one who really wanted these tapes," Gillooly says. "Because it gave her some sense of a family."

It was Jeannie, she explains, who was the motivator in making these recordings. "Jeannie was the one who would get the hotel room and lug the tape recorder – and those tape recorders were heavy. She would bring a record player, she would bring food, she would make sure the tape recorder was actually running when he knocked on the door so that she would get the moment of them greeting each other."

These happy scenes, the creation of a brief kind of normality, near-domesticity, are almost as heartbreaking as the more pained conversations. "But I think one of the worst is the tape of Jeannie where she is just so anguished and crying," says Gillooly. "That tape has been so highly edited and so compressed, but it was a really long recording. She recorded it at a slower speed and so it was more like 40 minutes long. Certainly you hear her break down and cry, but there was a lot more to it – how she felt about this desperate situation they were in."

Meanwhile, she says, "Tom could be quite pathetic. It was almost as if he wasn't listening to himself. He'd be contradicting himself all in the same tape. There's a tape where he says: 'I can't go on, I can't, I can't, I just can't do it, I can't do it' – many tapes like that where he has those frightful sad moments. And then two weeks later it's as if he doesn't even remember."

Tom never left his wife, though the affair ran for many years. In the course of her research, Gillooly even tracked down Jeannie, who was living in a nursing home, but able to tell her enough to make Gillooly appreciate the circumstances of the affair, suffering from dementia.

"She treated me as if I were in her parlour, making me feel comfortable and welcome," Gillooly remembers. "Though she didn't care who I was – she barely let me talk actually, I was just prompting some happy memories for her, for the most part. She was very personable, very sweet, very chatty, the way she is on the tapes, and though her voice is quite different, a very elderly-sounding voice, there were some phrases, a few words, that she had used on the tape. But the conversations were circular, they didn't necessarily make sense."

She asked about Tom, and was surprised to find that when Jeannie spoke "he was this professional man that she admired … she didn't talk about him romantically at all. When I asked about being in love, she would talk about her first husband." Jeannie, Gillooly discovered, was widowed young, her first husband killed in the war. "It did really make me understand so much more about her. That she really was trying to replace him."It is hard not to wonder how the experience of making this film affected Gillooly, whether it changed how she herself viewed relationships, the needs and desires of others, but she insists not. "There wasn't anything surprising in this at all to me," she says. "You can't help but compare it to what's happening now — to how people are exchanging these private moments today. But where it differs radically is that this couple were inventing the form. The significance of this material is really that it was so new – the technology was so new and what they were doing with it was unique. It was that time-capsule thing that fascinates me so much. We all think that we have witnessed reality shows in a way, but what's so startling about these tapes is they really do feel real. This is what's so astounding about it. It's so real, and it's so raw."

• Suitcases of Love and Shame is at The Art of the Real film festival in New York in April. Finding Vivian Maier is on release in the US and will be released in the UK later this year