Overloaded with information? Well, here’s a statistic to further clutter your mind: scientists at the University of Southern California have just calculated that the total amount of data stored in the world around 2007 was 295 exabytes. That’s 295 with 18 zeroes after it. Burn all that information on a compact disc and the resulting stack of CDs would reach beyond the moon. Print it out and you cover the entire surface area of China with 13 layers of books. But that was all way back in 2007. Even more mind-blowing is the rate at which data accumulation keeps accelerating. Today’s best guess is that 90% of the world’s current information was created within the past two years.

This torrent of knowledge is easy to explain. The costs of capturing, processing and sharing data have plummeted; mobile phone usage, electronic transactions and social media graphs have skyrocketed, unlocking an exponential treasure trove of behavioural data in the process; and companies have woken up to the realisation that all those real-time digital trails we leave behind us are the raw materials of tomorrow’s information factories. The result is a world gorging on data in the hope of turning those information streams into rivers of gold.

“Data is the plastic of this new New Economy,” suggests Om Malik, founder of GigaOM, the technology information house that just hosted a ‘big data’ conference in New York. “Just like polymers could be shaped into anything new, data can be shaped into providing better business decisions and creating new experiences.” Indeed, what has come to be known as ‘data exhaust – those customer insights hidden inside corporate vaults as the informational by-products of constant interactions with the marketplace – may hold greater economic value than a company’s core activity. They become the central asset. Silicon Valley futurist Paul Saffo says: “Many companies will suddenly discover that their main business is data.”

Not that this tidal wave of data bits and bytes has made the planet that much smarter. Quite the contrary. Using brain scans, researchers at Philadelphia’s Center for Neural Decision Making have been able to show how our reasoning abilities start slowing down or seizing up when faced with too much information. That ceaseless influx of tweets, texts and emails throws us for a mental loop. At best we give too much weight to the latest piece of data thrown at us; at worst, we become so tyrannised by all that choice that we lack the conviction to act at all.

The same brain-freeze applies to organisations: the more that companies drink from that fire hose of information, the poorer their decisions. “As computers get faster, organisations get dumber,” says JeffJonas, chief scientist at IBM’s Entity Analytics Group and an IBM Distinguished Engineer. At least, that’s been the case during this transitional period, in which data is being warehoused faster than the ability to make predictive sense of it all.

“Our decision-making may get worse before it gets better,” agrees Michael Driscoll, co-founder of Metamarkets, a real-time data start-up that provides price data and predictive analytics to large- scale global media companies.

Driscoll compares the current situation to where the US’s Central Intelligence Agency found itself in the 1970s when operatives started deploying reconnaissance satellites. “Instead of having just aerial photographs of places that they would target, they had satellite data for most of the globe. And the result was analysis paralysis.

Many of the analysts at the CIA had never been thinking of how to process all this data. So they drowned. In fact decision-making at the CIA got worse for about a decade. Which is where the emphasis on human intelligence comes in: knowing where to look, not simply trying to look at everything.”

The signal-to-noise ratio in analytics is so great that technology giants have spent at least $15bn these past couple of years snapping up data specialists that can help bring quick meaning to the cacophony. With corporations seeking ever better ways to divine trends, correlate relationships and make better predictions from their data hoards, business intelligence is now a $100bn industry and growing at almost 10% a year.

Last year, for example, IBM paid $1.7bn for the Massachusetts-based Netezza Corporation. On behalf of clients that include NYSE Euronext and Estée Lauder, Netezza designed a data-warehousing appliance that handles complex analytic queries at faster speeds than traditional systems. “The deal with IBM was representative of something that everyone is figuring out, which is that analytics based on big data really matters,” says Netezza’s CEO, Jim Baum. “This emerging industry – big data and the predictive analytics associated with that data – is going to create more business value, more ROI for industry at large than any of the other big technology trends that you are seeing. It is that big.”

As Netezza’s acquisition underscores, the rewards from this bonanza-in- waiting will go to those who can deliver business intelligence the fastest. “If I have to wait three days as a human being to get an answer, I am not going to ask the next question. But if I can get an answer just like that, my creative mind takes over and I say: ‘Aha, and what if I ask this question and then that next question?’” says Baum. He also cautions that “the human ability to ask all the right questions is limited. At some point the data has to find the data and the relevance has to find you.”

But right now there is a disconcerting trade-off between analysing large volumes of data quickly versus spending the time to draw the soundest, most accurate conclusions from that data. Between the two, speed wins out in this real-time economy – even if it was that microsecond agility that blinded the banking world to its recent meltdown risks. “The decisions are worse [than those made during the offline era] but their timeliness is so much more valuable than their correctness,” says Theo Schlossnagle of OmniTI, a global IT services company involved in web applications, database services and internet architectures.

Among those who grasped this pent-up need for data analysis on the fly is Bill McColl. A former professor at Oxford University, where he headed the Parallel Computing Research Centre and chaired the computer science faculty, McColl left British academia in 2006 for Silicon Valley, where he went on to found Cloudscale with $1.2m in angel funding. Claiming to be the world’s fastest big-data solution, Cloudscale’s platform can analyse a live stream of information in real time at more than 150 megabytes a second, a speed suited for such applications as algorithmic trading, fraud detection, mobile advertising, location services and marketing intelligence.

Such performance is 125 times faster than Yahoo’s recently released S4 (Real-Time MapReduce) system, on the same hardware – about the difference in speed between walking from San Francisco to New York versus taking a plane. Such cloud-speed records are more than just a matter of bragging rights. “These days, data has to be looked at as soon as it is generated. The stream never stops and you don’t get a chance to catch up if you fall behind. In some scientific areas you may be able to go away and gather three months of data and crunch on it using MapReduce and other types of tools. But in the business world, where you want to act on that data, you’ve got seconds – and if you don’t keep up with it, your business is toast,” warns McColl.

“The killer app remains real-time commerce. Every large enterprise, all web companies, everyone who runs large websites, all now want an instantaneous, 360-degree view of everything that is happening in their business. Companies like Google that have really pushed the envelope on a lot of things have shown that the more data you have and the more smart you are at extracting knowledge from that, the more competitive you’ll be. And that’s now spreading into every enterprise.”

Exciting as this might be for techies, there’s a good chance that ‘data processing’ throughout the general business world still basically amounts to looking things up on an Excel spreadsheet. Which is why McColl has also just introduced Cloudcel, a plug-in that transforms Excel into a cloud-enabled application where data can be dropped live into a spreadsheet that can perform hefty analysis. Imagine you could sit in Excel, McColl explains, and with one click launch an app on billions of rows of data that would normally take days to compute. And you get the answer back in less than a minute.

This ability to put complicated data analysis into the hands of normal business users is seen as a crucial next step. Company executives will always feel more comfortable taking charge of their own customised computations, sliced and diced according to their instant needs, rather than have to wait upon IT departments to deliver the results. In fact they will try to avoid IT where they can. “Part of the missing link is that the data technology tool-set has not been better integrated in the enterprise. Data decisions are best managed by the business,” insists John Lucker, who leads Deloitte’s Advanced Analytics & Modeling Practice. “The business needs to focus on its competitive advantages. And the only way in which significant advantages can be created from this data is if the technology is taken out of the hands of IT and control of the tools given over to those with the business insights.” Even then, there are behavioural challenges to confront. Executives still need to be persuaded to put aside their entrenched tribal wisdoms and also to overcome any confirmation bias – that human tendency to zero in only on facts and figures that corroborate one’s own worldview.

Helping to bridge such divides is an emerging class of highly sought-after professionals that have come to be known as ‘data scientists’. Left to their own devices, suggests Lucker, your typical computer programmers will keep “shining their stones” even if those last few, time-consuming algorithmic refinements yield ever-diminishing returns. But this emerging breed of corporate number crunchers and data wranglers has the skills not only of mathematical statisticians and computer programmers, but those of business-savvy pragmatists who understand the commercial imperatives of a particular enterprise. The ability to present data in meaningful ways that can be easily visualised and acted upon by management is another critical skill. Not surprisingly, says Lucker, there is “an enormous talent shortage” in this regard.

Once companies are properly set up to leverage their real-time data signals, an endless array of opportunities and spin-offapplications present themselves (see box). And so also do the privacy headaches. On 1 April, of all days, a Dallas-based firm that sends out 40 billion emails a year on behalf of clients such as JPMorgan Chase, Citibank and Target, disclosed a giant security breach. At worst, this particular attack on email-marketing company Epsilon will probably result in a surge of online spam and ‘phishing’ attacks – fake emails designed to steal information such as account numbers and passwords. But it has already raised uncomfortable questions in the public’s mind about how much of their private data should be stored in the cloud – the same data that companies have come to rely on for customer insights and personalised product recommendations.

The extent to which marketers can build a picture of our daily lives became evident in Germany around the exact same time as the Epsilon breach. Malte Spitz, a Green Party politician, caused a global stir when he used German privacy law to force his mobile phone carrier, Deutsche Telekom, to divulge what it knew about him. The result, published in Die Zeit, exceeded even the most paranoid expectations: nearly 36,000 pieces of information that detailed his exact whereabouts during a six-month period. And that was just the GPS information gleaned whenever Spitz checked his email. In reality, phone companies check phone signals every seven seconds in order to determine the nearest tower, enough monitoring to unravel anyone’s secrets when triangulated with other information sources.

Although in this case, the data in Spitz’s case was only mildly embarrassing – it showed that the environmental champion sometimes took a flight rather than a more fuel-efficient train – the wider implications are more serious: not only can all our movements be traced, our intentions can be deciphered too. According to IBM’s data guru Jonas, the roughly 600 billion data transactions that are recorded from cellular phones on a daily basis in the US can help predict where someone will be on a given day and time with up to 87% accuracy. “A surveillance society is not only inevitable and irreversible, it’s worse: it’s irresistible. And you’re the ones doing it. You want the location-based services, you are signing up for the free email. You read the MOU and some of them will say: ‘All the data is ours. If you quit and delete your account, we get to keep it. It is still ours.’”

Anticipating the data privacy battles ahead, LinkedIn founder Reid Hoffman had this advice to give in a keynote speech at this year’s SXSW interactive conference in Austin. First, “Never ambush your users. If they trust you, they should never feel ambushed by what you do with their data.” Second, “Not all data is created equal.” A lot of the data that people provide online isn’t harmful — what is damaging is “data that is essentially equivalent to a password”. One way or another, data will change all our lives. That much is a no-brainer.

THE GAME’S AFOOT

Earlier this year an IBM super-computer known as Watson defeated the two finest human quiz-show champions ever to play the popular American TV game Jeopardy.

This was no trivial feat. Beating a chess grandmaster is one thing, but creating a machine that can handle natural language questions with all the attendant ambiguities of meaning and subtleties of context had come to be considered a futile exercise in artificial intelligence. Puns, anagrams, Shakespearean wordplay, all had to be ‘understood’ by Watson in fractions of a second. It also had to assess risk – since winning depends on knowing how much money to stake on any particular answer.

Watson, named after IBM’s founder rather than the fictitious sidekick to Sherlock Holmes, took home $1m in prize money.

But he won’t be enjoying an early retirement playing virtual golf. Watson’s creators have identified several potential applications for his superhuman skills, including online tech support, traffic-pattern optimisation and the ability to parse vast tracts of legal documents to unearth details that might otherwise elude lawyers.

But the biggest payoffmay well be in the field of medicine. According to American Medical News, IBM has partnerships with eight major universities to supply medical data for Watson’s information base. While it might take up to 30 years for a doctor to process such data in order to perfect their evidence-based diagnostic skills, Watson ought to be able to suggest appropriate treatments based on a list of symptoms fed into its computing cores. IBM will also be testing its bedside manner by seeing how patients might react to speaking to this virtual Dr Watson.

Healthcare is just one of the areas in which IBM sees dollar signs in all that data. At a microchip plant in Vermont, for example, the company has wired all the pumps, tanks and pipes with 5,000 electronic sensors in order to measure the water flowing through the system in terms of temperature, flow rate, pH level and clarity. The benefit stretches beyond the need to create ultra-pure water, while saving on water usage. Armed with that stream of 300,000 data points a minute, IBM is able to create nine custom varieties of water – selling each for four, five, even 10 times the cost of buying that water from the municipal authority. Furthermore, IBM believes that selling such smart-water information technology could be a $15bn- $20bn annual business in of itself.

Keeping up with the flood of information is clearly big business, particularly if that data is predictive. Just ask AC Milan. For a couple of years now, the Italian football club has relied on business-intelligence software to anticipate sport injuries and prolong the playing life of their expensive team. Every fortnight, members of the team perform an elaborate series of tests, including a jumping drill that measures the angles of their knees. In the first full season of assessing such data, player injuries declined by two-thirds.

The biggest trophy, however, will go to those who can use data to perfect recommendation engines in such a way as to read the customer’s mind in real-time. “Customers love ‘next best offers’ provided those offers are insightful and relevant. They want their time to be better spent,” says Deloitte’s data analytics expert, John Lucker. “The biggest opportunity will come from creating more meaningful touches with our customers – using real-time data to assess where individuals are their life cycle and then instaneously steering the conversation or transaction in a way that leaves a positive impression and results in higher sales. Coupling those predictive signals with the most relevant business context would be something like a business Nobel Prize.”

For several years now, Google has talked of a world in which it not only organises the world’s information but delivers it to customers before they know they need it. Its former CEO Eric Schmidt even went so far as to say Google users should be able to ask open-ended questions such as ‘what shall I do tomorrow?’ or ‘which job shall I take?’ In other words, dumb machines that make humans smarter. Until Watson came along, such analytic capabilities seemed hopelessly far-fetched. Now they seem almost elementary.