Jeff Bezos likes to surprise. Roaming Amazon’s global headquarters in 2013, the tycoon promised a television crew half his fortune if they could guess his company’s latest innovation. They did not.

“Oh my God,” one of his wide-eyed guests exclaimed, as they caught sight of autonomous delivery drones.

Bezos, a self-declared optimist, suggested it could happen by 2017, or maybe 2018. “I know this looks like science fiction. It’s not,” he told 60 Minutes on CBS in 2013. Ten years later, in all but a limited number of locations across the world, it has yet to become a reality.

Yet, this week Amazon announced it is expanding its drone delivery service to the UK and Italy and growing its service in the US. Amazon is just one – very big – player that believes a breakthrough is within reach.

Progress has clearly been slow. But operators facing stringent regulatory hurdles have been quick to talk up the scale of their small-scale programs. Walmart, another retail giant, claims to have completed more than 10,000 drone-based deliveries since first looking to the skies two years ago. It saw fit to mark the milestone this summer by dropping a vast Oreo cookie into a 6ft cup of milk. Oreos were the top items delivered by a Walmart drone.

Amazon says its Prime Air service has delivered “thousands” of packages. It is now preparing to expand the program beyond the US for the first time, having this week announced plans to launch in the UK and Italy by the end of next year.

“Over the next couple of years, we will see drone delivery networks really starting to happen,” said Andreas Raptopoulos, founder and chief executive of Matternet, one of many operators in this nascent, yet crowded, industry.

Companies from Alphabet, owner of Google, to multibillion-drone startup Zipline are jostling to prove their aircraft amount to the future when it comes to delivery of light items, from toothbrushes and takeaways to medicine and toys. Alphabet’s Wing venture, which has been operating in Australia for several years, and also has a deal with Walmart, claims to have completed no fewer than 350,000 commercial drone deliveries. Zipline, which initially concentrated on Africa, says it has completed almost 800,000.

So far, the locations of many of such programs have been carefully-picked and often relatively sparse. Despite the bold dreams and proclamations of its deep-pocketed pioneers, ultra-fast shipping has taken an awfully long time to deliver.

Should Amazon start dropping off packages from autonomous aircraft in the UK by the end of 2024, as promised, it will be eight years after the tech giant announced the completion of its first commercial drone delivery, in Cambridge. This early test only heightened the buzz around the supposed next generation of e-commerce, until the company was reported to have scaled back the UK arm of its Prime Air service in 2021.

Those seeking to clear the sector for take-off nevertheless claim it is gaining altitude. Raptopoulos envisages fleets of between 250 and 500 small drones dropping off packages across locations like Mountain View, California, where Matternet is based. “We think we’re going to see that type of thing happening certainly in this decade, and probably in the next one to three years.”

This optimism is fuelled by the belief that aerospace regulators and policymakers are increasingly likely to permit the scale-up of commercial operations. The UK’s Civil Aviation Authority, for one, says it is working to make drone flights beyond the visual line of sight – where human “spotters” are not required to constantly watch the aircraft – an “everyday reality”, after ministers pledged to make them “commonplace” by 2030.

America’s Federal Aviation Administration (FAA) has, since August, authorized several companies to operate drones beyond the visual line of sight. These include UPS, which is using Matternet’s M2 aircraft to deliver small packages.

The Biden administration has tapped Mike Whitaker, a veteran aviation executive, to be the next head of the FAA. During a Senate hearing on his nomination earlier this month, Whitaker said the US must “build the aviation system of the future” to account for drones and flying taxis, adding that this would require the agency to “be forward-looking, adapt quickly and execute a plan for the future”.

Regulators appear to be picking up the pace. Raptopoulos had first expected authorization for Matternet’s vehicle to fly beyond the line of sight in 2020, three years before it was granted. “The steps have been happening,” he said. “It’s just that the pace has been slower than, I think, [the FAA] would have liked, and the industry would have liked.”

Certain regulatory “steps” would fundamentally transform the economics of drone delivery. A pilot at shipping giant UPS is now authorized to control up to three aircraft in the US, for example, but Raptopoulos is looking forward to that ratio rising to “20 to one, or 50 to one, or 100 to one. Because these systems are built to be highly autonomous.”

The ultimate challenge will be reassuring the public that these vehicles are safe and reliable. And while their makers stress the noise emitted during flights is now akin to that of a lawnmower, some customers already receiving deliveries (and, presumably, their neighbors) say they remain loud.

Amazon too now advocates the benefits of patience. “We’ve been at this for a while,” said David Carbon, who leads Prime Air. “We will move as fast, or as slow, as the communities and the environment allow us to.”

It is easy to see why regulators – and the general public – have their doubts about such a big change in how we deliver goods. But there is no doubting that home delivery is here to stay – especially post-Covid. And within the sector, some have grown a little tired of explaining why they see flying packages to homes – often one at a time – as an efficient, workable means of serving millions of online shoppers. Does it really make more sense, they ask, for a two-tonne car to carry a three-pound package to your doorstep?

Carbon points to the British capital. “No one really wants cars in London,” he said. “Now there’s lots of cars in London, right? But this technology really ensures that our customers can get what they want in under 60 minutes without having to put themselves in an inconvenient state.”

Senior figures at Amazon like to cite the late futurist Roy Amara, who said that a technology’s effect is overestimated in the short term, but underestimated in the long.

Delivery drones have so far failed to live up to the hype. Undeterred, their developers – still battling to prove that unmanned aircraft have a viable place in the future of shopping – are looking to launch fleets into suburban skies. Is your neighborhood ready?