Light travels at 186,000 miles a second in a vacuum, which is another way of saying that it covers 186 miles in a milli-second – a thousandth of a second. Given that much of our contemporary electronic communications are conveyed by pulses of light travelling along fibreoptic cables, we are given to extravagant hyperbole about the "death of distance". After all, if a message – or a file – can traverse the globe in the blink of an eye, it doesn't matter whether your hard drive is on your desktop or in a server farm in Nebraska or Sweden.

But it turns out that the speed of light is of great practical interest to some people. One group of them have shelled out $300m to lay a fibreoptic cable in a straight line from Chicago to New York. This involves, among other things, drilling through mountains and under urban areas. And for what? So that the time taken to send a signal between New York and Chicago could be reduced from 17 milliseconds to 13. For that apparently infinitesimal improvement, stock market traders were willing to pay $14m a year, plus a substantial upfront payment, to use the cable.

Therein lies the tale of Michael Lewis's enthralling new book, Flash Boys, which joins an elite but growing list of volumes that set out to explain how computing is reshaping our world. For it turns out that for people armed with the right kit, software and networking skills, an advantage of a few milliseconds is enough to let them turn a $14m annual subscription into annual profits of $20bn.

These people are called high-frequency traders – "high-frequency" because they are incredibly active (they submit almost 99% of the orders on US stock markets) and buy and sell shares in milliseconds. They are not really "traders" in any normal sense of the term, but software algorithms, and they now dominate the most important stock markets in the capitalist world.

Most of us knew that, sort of. We knew about computerised trading and probably naively assumed that it was more efficient than the old system of guys in coloured jackets bellowing at one another on the floor of an exchange. Well, it is more efficient – for some. But the significance of Lewis's book is that it explains in user-friendly terms how the colossal profits of high-frequency traders really amount to an unconscionable tax on the ordinary investor, or at any rate on the pension funds and other financial institutions on which our livelihoods depend.

Flash Boys follows the usual Lewis formula: find a scandalous situation that is too arcane for most people to comprehend; locate some smart guys (they are usually male) who have spotted the scam and plan to do something with or about it; and tell their story.

In his earlier book The Big Short, Lewis focused on the smallish group of shrewd investors who understood that the sub-prime mortgage boom was sure to go bust and bet against it. In his new book, the white hat is worn by Brad Katsuyama, a Royal Bank of Canada trader who discovered the extent to which high-frequency traders were skewing the stock market and screwing investors and, in the end, set up a new stock exchange (IEX) designed to level the playing field by making sure everyone's trading instructions arrived at the same time.

But the most interesting thing about Flash Boys is what it reveals about the networked world into which we have stumbled. Once upon a time the New York stock exchange was a place; now it's a set of more than a dozen "stock exchanges" scattered around New York and New Jersey. But these exchanges are not places either: they are server farms, air-conditioned warehouses filled with rack-mounted computers, complete with blinking lights and whirring discs.

So the stock market has become a virtual space – an interactive, computer-driven system of staggering complexity. And it turns out that there are several sides to this complexity: for the banks and the high-frequency traders who exploit it, it's a marketing tool for bamboozling investors and a means of intimidating regulators; and for smart programmers and entrepreneurs it offers limitless opportunities to play the system.

"From the point of view of the most sophisticated traders," Lewis writes, "the stock market wasn't a mechanism for channelling capital to productive enterprise but a puzzle to be solved."

For society, the system's complexity is dangerous, because complex systems are intrinsically unpredictable and nobody really understands this one as a whole – which means that catastrophic failure is always a remote but finite possibility.

At first sight, the modus operandi of high-frequency traders seems so outrageous that one assumes it must be illegal. In its review, the Economist came up with a useful everyday analogy: high-frequency traders are like "the people who offer you tasty titbits as you enter the supermarket to entice you to buy; but in this case, as you show appreciation for the goods, they race through the aisles to mark the price up before you can get your trolley to the chosen counter". How, one wonders, can it be legal for a handful of insiders to operate at faster speeds than the rest of the market and, in effect, steal from investors?

But it is legal, and for an interesting reason. In 2004 the US Securities and Exchange Commission discovered that some traders in the old New York stock exchange were exploiting the discretion then allowed to them in choosing the time to execute a deal. The following year the SEC passed a new regulation, known as Reg NMS, which obliged traders to seek "the best price" for a security.

What the SEC did not anticipate was that in the new fragmented system of a dozen virtual exchanges, this provided the opportunity for high-frequency traders to outrun the market while staying within the law. Reg NMS was a well-intentioned measure to restore equality of opportunity in the US stock market. But instead, as Lewis points out, "it institutionalised a more pernicious inequality. A small class of insiders with the resources to create speed were now allowed to preview the market and trade on what they had seen."

This is a good illustration of one of the central problems that society will have to address in the coming decades: the collision between analogue mindsets and digital realities.

Software is pure "thought-stuff". The only resource needed to produce it is human intelligence and expertise. This has two implications. The first is that attempting to regulate the things that it creates is like trying to catch quicksilver using a butterfly net.

The Edward Snowden disclosures about the US National Security Agency have revealed how difficult it is to bring this stuff under effective democratic control. Lewis's account of how high-frequency trader geeks have run rings around the regulators suggests that much the same holds true in civilian life. This technology can easily run out of control.

The second implication is that what one might call the politics of expertise will become much more important. Mastery of these technologies confers enormous power on those who have it. Sed quis custodiet ipsos custodes and all that. So in addition to wondering who will guard the guardians, we may have to start thinking about who is going to guard the geeks.