The games industry has a glorious future planned for us. It is a future of seamless co-operative and competitive gaming, where every single-player adventure will magically segue into multiplayer face-offs; where your friends will be able to drop into your game world at a moment's notice and join you as you fight dragons or attack battleships.

Activision's forthcoming space epic Destiny, likely to be one of the biggest releases of this generation, is entirely built around this whole idea of community interaction. But it is far from alone – Watch Dogs has drop-in competitive matches, Far Cry 4 will allow instant online co-op, while Forza Horizon 2 and Sunset Overdrive both tempt you in and out of competition with friends and strangers from within the main campaigns. This is the way things are going.

There's just one problem: plenty of gamers don't want it. Plenty of gamers, even in this age of cloud computing and continuously connected games consoles, just want to play alone. The industry may be mystified by these digital era luddites, but there are good reasons for their intransigence: they represent a whole way of thinking about what games are and the experience they provide.

Everyone was alone in the olden days

Partly, of course, it's a preference that's steeped in the history and tradition of gaming. The chances are, if you played games before 2000, you mostly played alone by default – or you occasionally allowed one friend into splitscreen sessions.

"I grew up in the 1980s when single player games were – with few exceptions – all there really was," says veteran gamer Dave Kirk. "Maybe that's a subconscious influence on me even today. I really liked Grand Theft Auto V, and I'm big-time into racing games such as F1 2013 and Forza Motorsport, but I play all of them exclusively single-player."

Talk to gamers over 30 and they'll often be nostalgic for classic adventures like Secret of Monkey Island, Deus Ex and Final Fantasy, games that wrapped you in a world for many hours – days even – and allowed you to wallow in them alone, like good books or epic TV series. A great single-player game is the perfect solipsistic fantasy, it's you by yourself, saving the world – and everything revolves around you, not a guild of friends, or an army of strangers.

Many players still want those epic all-consuming narrative experiences, and that sense of personal responsibility. "The earliest I remember was Wing Commander on the Amiga," says Simon Matthews, a PR manager based in Egham, Surrey. "It really felt like I had an impact on the story and the game; I remember feeling upset and angry when Iceman died, but I had the sense that it was my fault for failing missions. Not enough games let you fail anymore."

There are gamers, then, who play games as a form of pure escapism – not just in terms of escaping to new worlds, but also escaping other people. "I prefer to play alone," says April Pereira-Finn. "I'm very people-focused at work and it's nice to switch off for a few hours, weld my jaw shut and get into a great story ... I'm a single parent, too, but when it's bed time for my son, it's game time for me."

"I don't have the time to get good at Call of Duty"

This need to condense gaming sessions into narrow periods of valuable free time is another reason why plenty of people – especially those with kids and careers – want to play alone, in their own way, at their own pace.

"Having a job and an adult life means I can't start playing a game on the day of release, and be up to level 45 by the end of that weekend," says Richard Hancock, a local radio producer. "Sticking with the single-player, finishing it, and then moving on to the next game also allows me to play a wide range of titles, rather than spending all year playing the same game, just waiting for the next map-pack to freshen things up."

Another big factor is alienation from the online gaming community. Multiplayer shooters particularly are dominated by young men trash talking each other, often with sexist, racist and homophobic slurs. The experiences some people have had while trying games like Call of Duty and Halo online have put them off forever.

"Playing with jerks is no fun," says internet cartoonist and keen gamer Jeffrey Jaques. "Having a stoned college kid tell me I suck and can’t fucking shoot when I’m just learning an FPS is no fun. Being stuck at level 40 in World of Warcraft when all my friends are level 60 and don’t have time to help me catch up is no fun. Having to play any multiplayer game for 25 hours a week to remain competitive ... that's not fun, that's an unpaid part-time job."

The money behind multiplayer

So if there are plenty of gamers who still want to play lone campaigns, why is game design orthodoxy moving toward shared narrative experiences? Is it just that a majority of people welcome our seamlessly connected future? Perhaps. But we shouldn't rule out the commercial imperatives.

Multiplayer games are easier to monetise after release, allowing developers to sell extra downloadable content like maps to fight in, and new weapons and scenarios. They also encourage players to buy in early – because everyone wants to get good at the game as quickly as possible – which means a nice flood of upfront sales.

"With boxed products, I think it is because assets are so damn expensive," says industry analyst Nicholas Lovell, author of The Curve, a book on the future of the video game business. "Multiplayer can keep gamers playing for a much smaller underlying cost than single player, particularly when it is narrative led. And multiplayer has viral acquisition techniques, which reduce market costs, and viral retention techniques, which keep players in your game, and not your competitors'.

"[A focus on multiplayer] can also lead to microtransactions or other advertising revenue. It enables business to switch, as I believe nearly all businesses need to, from being volume-led (how many did I sell) to ARPU-led (how much money am I making from this customer, and do I keep satisfying this customer so he or she stays with me)."

The future is social

So are we looking at the end of pure single-player experiences? Are the gamers we spoke to facing a future in which their way of interacting with games will no longer be catered for? It's unlikely. Hopefully. The recent success of The Last of Us and Grand Theft Auto V, both games that appealed mainly due to their campaign content, shows that there is still a mass market for lone narrative experiences.

The growing indie sector is also supplying a steady stream of highly subjective single-player titles from Dear Esther and Gone Home to FasterThanLight, to Telltale's episodic adventures, Walking Dead and Wolf Among Us.

The latter examples hint at where the traditional narrative experience is going. In Walking Dead, each instalment closes with some data on how every other player made their decisions.

In the coming years, we're likely to see a new era of campaign modes, which can be played alone but are enhanced with social elements. We'll be able to record and share key moments in our adventures, and we'll have seamless online venues to talk about what we've seen.

Indeed, more titles are now using procedurally generated landscapes and narrative elements, which supply every player with unique experiences, formed on the fly just for them – games like No Man's Sky and Fortnite. Through this technology, certain games will naturally become both more insular and individual, but also more open to sharing. We will see things no one else will.

"Developing and supporting a multiplayer mode is expensive and not all games suit this type of play," says Piers Harding Rolls, head of games at industry analyst IHS Technology. "So, actually, although all content is becoming connected, I believe the new social and community features we see – streaming via Twitch or uploading to YouTube for example – offer publishers an alternative engagement feature set beyond the multiplayer mode.

"I think single player games still have a strong role to play in the future."

• The five most disruptive ideas in video game design

• Why gaming needs its Game of Thrones moment

• The identity paradox: why game characters are not us, but should be