Experts have warned of more internet blackouts after a crash disabled eBay in northern Europe this week, as the decades old systems used to organise the world wide web creak under the sheer weight of new devices and websites.
The web passed a milestone in its relentless growth on Tuesday: traffic across a key element of the internet reached unprecedented levels and caused meltdown for some of the older pieces of equipment which keep information flowing.
As engineers around the world rushed to fix the problem, one internet traffic monitor reported that the number of routes suffering outages on Tuesday had doubled to 12,600. On a typical day outages normally affect 6,000 routes.
Complaints began to flood chat forums and Twitter as eBay auctions ground to a halt and office workers found themselves locked out of Microsoft's cloud service Office 360 or unable to use the password protection service LastPass. "The internet just broke under its own weight," said technology website The Register.
At the heart of Tuesday's problems was a system known as the Border Gateway Protocol (BGP) – essentially a map of the internet that lists hundreds of thousands of routes or pathways connecting laptops, tablets and smartphones to each other and to websites and company servers worldwide.
BGP lists routes for the external web, connecting networks owned by broadband providers such as BT or Virgin, linking them to each other, to broadband providers in foreign countries and to destinations such as Apple or Google data centres.
This week the number of routes mapped by BGP passed 512,000. The number is significant because some older routers – the machines which form the backbone of the internet by selecting the pathways taken by packets of information – are not able to remember more than 512,000 routes. Beyond that number they begin to slow down or simply forget routes, causing outages.
Some web watchers are blaming the rise in traffic on an apparent mistake by the giant American broadband company Verizon, a problem which lasted only five minutes but was big enough to bring down large swaths of the internet.
Vancouver-based firm BGPMon, said most of the larger internet companies were around the 500,000 mark - each provider has a slightly different list of routes. Level 3, which joins up national internet companies, was at 498,000 on Thursday.
"Now that we are so close every minor mistake tips us over," said Andree Toonk at BGPMon. "In a few weeks we will organically go over that 512 limit and we will probably run into a similar situation again. There will be outages and certain destinations will become unreachable. For a lot of people in the industry this was a wake up call."
Toonk says the problem began on Tuesday at 8.48am UK time, when a flood of 15,000 new routes was introduced. He found almost all the new routes came from Verizon, which connects millions of American homes and businesses to broadband.
Each route is a gateway giving access to thousands of individual IP addresses. But instead of announcing a few new routes aggregating packets of thousands of addresses, Toonk said Verizon announced lots of new routes aggregating small blocks of just a few hundred addresses, and that this was probably a leak from its internal system.
"Old hardware that is at least five years past its end-of-life sulked, because it ran out of memory," explained James Blessing, chair of the Internet Service Providers Association, which has close to 300 members across the UK.
The US technology giant Cisco has been warning since May that some of its older routers would collapse under the strain unless they were given a software update. But updating or replacing them can be complicated. "You can simply change some values on the boxes and then restart the entire machine," said Blessing. "Unfortunately these boxes have hundreds of customers attached to them so getting permission from them all to do that is a pain."
Without the update, however, another traffic spike will turn these Cisco routers into "really expensive doorstops", as one IT expert warned.
But a bigger, related problem is looming – the world is running out of internet addresses. National broadband providers have been slow to move away from the current system that assigns an internet address to each device, much like telephone numbers. Known as IPv4, the system has 4.3bn potential combinations, but the next generation, IPv6, which uses more digits and combines them with letters, will have a potential 340 trillion trillion trillion – more than the planet is ever likely to need (or at least for a while).
Some parts of America have already adopted the new address system but the UK is playing catch up. Sky, Virgin and BT have already issued some wifi boxes to their residential customers capable of using IPv6, and Virgin is running both systems in parallel, but none of the companies has entirely made the switch.
"We are reaching breaking point on this area of handling internet addresses," said James Gill, chief executive of GoSquared, an internet traffic measurement group. "These outages will continue happening because of the growing number of devices on the web."
