Sci-fihas long wrestled with the notion of the all-powerful computer. From the mysterious HAL 9000 in 2001: A Space Odyssey to Deep Thought, the machine tasked with finding the meaning of life in Douglas Adams’

Hitchhiker's Guide to the Galaxy, computers that represent a superior form of intelligence have captured our imagination. And such computers are edging closer to reality.

As far back as 1997 – an age ago, in computer terms – IBM's Deep Blue beat world chess champion Garry Kasparov over a six-game match. In February, another IBM computer, Watson, beat some of the world’s brightest humans in the general- knowledge TV show Jeopardy. Watson’s prize came to more than three times that of the next placed, human, runner-up.

Both were incredibly expensive research tools, tailor-made for a specific task. But behind the scenes, the power of such computers is being harnessed for tasks ranging from climate- change modelling to pricing securities and detecting fraud.

The new generation of supercomputers is helping to design racing cars, and to make family cars safer. They are developing new formulae for medicines and can translate, summarise and index texts. And high-performance computers could soon fly aircraft.

POWER SURGE

The early supercomputers were the preserve of governments, academic research institutes and the military. Computer scientist Dr Frank Baetke recalls working on Germany’s Cray-1 project as a physics researcher at the Max Planck Institute in the 80s. “It had eight megabytes of RAM,” he says. Today, a mid-range laptop has 500 times as much memory.

Two developments have changed the world of supercomputers – or as it is also known, high-performance computing – and brought it within reach of businesses.

The first is Moore’s Law, the theory that the power of a computer doubles roughly every two years. Moore’s Law has drastically driven down the price of key computer parts, such as central processors, and memory. According to Stephan Gillich, Intel’s EMEA director for high-performance computing, in 1997 a gigaflop of computing power cost $55,000. By 2010, it cost just $100.

The second factor is gaming. The growth of computer games as a pastime has fuelled an entire industry of components and technology designed for more realistic play, but which are now being put to use in areas as diverse as aircraft design and financial markets. IBM has developed a 3D emulator for aircraft design and maintenance using PlayStation 3 games consoles; the PlayStation itself shares many technical characteristics with a supercomputer. “We are seeing a crossover from entertainment and gaming to high- performance computing,” says Phil Dawson of industry analysts Gartner.

Software designers, meanwhile, have captured the power of graphics cards from manufacturers such as AMD and Nvidia to turn their chips, designed for 3D games graphics, into powerful machines for solving complex equations. Recently, AMD demonstrated a chip comtaining 400 of these specialist processing cores, but which is small enough to go into a laptop. The biggest computer in Japan uses Nvidia graphics processors – similar to chips on sale in a high-street computer store – for much of its power.

“The tendency to move from 2D to 3D gaming, and to use a PC as the host, allows us to develop very, very advanced accelerator cards. These cards are made in high volumes and provide immense firepower,” says Dr Baetke, who is now HP’s global high-performance computing technology programme manager. “These can provide 50 times the power of a standard processor.”

This type of increase in power allows researchers to tackle problems which previously were seen as out of reach, such as mapping parts of the human brain or simulating the creation of a black hole. But in business, they are bringing a different set of benefits.

Business problems rarely demand the same absolute computing horsepower required by academic or government research. But what they do require is a timely answer to a problem.

Aprevious generation research supercomputer may have been able to model a financial without moving out of second gear, but it would have taken months to programme and days, if not weeks, to run the model. Timescales meant the results would be mostly of theoretical interest. Today’s supercomputers are cheap enough for large businesses to buy and small enough to fit under a desk, but above all, they are lightning quick. One French investment bank, which upgraded its technology, moved from a share-price prediction model it could run once a week to one that it could run every day.

More frequent pricing predictions allow banks to fine- tune their strategies, beat rivals through better pricing and develop a much better understanding of daily and intra-day trading risks. And highly advanced computers are giving rise to a new type of bank. Known as ‘flow monsters’, these are tier-one investment banks that make their profits on commodities and derivatives by trading on very small market movements and margins, but in huge volumes. “The only way to do that is to have a lot of automation in pricing, risk management, limit management,” says Gordon Mackenzie, a director in the financial services practice at Deloitte.

The next step is to look at real-time data streams, initially for suspicious or unusual trades. “The banks are dealing with millions of market prices and thousands of counterparties, so spotting problems is a real issue,” says Mackenzie. “Being able to see a build-up of risk, or of a large position with a counterparty, gives you a ‘heat map’ of the markets.”

Such innovations are not, though, without their own problems. The so-called ‘flash crash’ of 6 May 2010, which caused shares to collapse on US markets, happened because computers misunderstood a single large trade. According to the Financial Times, an investigation found that one investor’s quick action to hedge a position by selling futures prompted other banks’ automated systems to start selling, fearing a market downturn. Other systems then joined in until authorities, almost literally, pulled the plug. Regulators have since put in place new rules to stop wild swings being caused by computer trading. But the incident illustrates the danger of allowing computers to take charge. Computers, unlike humans, are unable to put what they see in context.

“You have trusted and untrusted data,” points out Andy Mulholland, chief technology officer at Capgemini, an IT consulting firm. “There is an inbetween stage of ‘trusted in context’. If you are in sales, and you know a competitor is running a seasonal offer, that is trusted in context. You adjust your pricing accordingly. But if you model your whole business using that data, that is untrusted.”

A specialist, whether they're a designer, doctor or financial trader, can spot unexpected outcomes or outlying data through inference and intuition. A computer has to keep crunching the numbers. And as computers cannot, for now, learn in the way that humans learn, they are only as good as the data that humans programme into them.

This is illustrated not just by the flash crash, but by the challenge of creating a truly autonomous robot, car or aircraft. Trials such as Google’s self-driving car, or BAE Systems’ Mantis, an autonomous aerial vehicle (rather than a remote- controlled ‘drone’) draw on the massive advances in computer power, as well as in sensors including radars and digital cameras.

Should they work, autonomous systems could save significant sums of money. Aside from defence applications, autonomous aircraft could be used for TV newsgathering, traffic surveillance, weather monitoring and telecommunications. They can be much smaller than manned aircraft and could stay airborne for hours on battery or solar power. But there is a way to go before such systems can operate in civilian airspace. Even supercomputers struggle with “unbounded” problems (moving targets), says Dr Stefan Wesner, managing director of HLRS, the high-performance computing centre at the University of Stuttgart. “In a supermarket, the [possibilities] for allocating staff are limited, and there is a lot of historic data,” he says, citing an increasingly common task carried out by computers. “But an aircraft has to be prepared for unforeseeable situations. If you can limit the decision space, it is much easier to control the problem, and ensure the data are sufficient.”

Not all problems, he warns, can be reduced to a neat mathematical algorithm, nor do all the problems researchers and businesses want to solve result in the type of yes or no answers computers excel at. Even if an aircraft designer could bring together the world’s best pilots to design the algorithms for an autonomous craft, there are no guarantees, Dr Wesner suggests, that they would even agree with each other.

Instead, supercomputers are being used to support human decision makers, to eliminate the unlikely and the uneconomic, and to allow us to test a far wider range of “what if” scenarios before making a final judgement call. This way, a car- maker can assess more designs in software, and so subject fewer real prototypes to physical crash tests; a pharmaceutical or chemical firm can reduce the number of formulae it needs to make up for lab or field trials; and a bank can carry out more Monte Carlo simulations to work out the risks of a trade, fine-tuning the numbers each time. “You have better design, by having more iterations of the design you are looking at,” says Intel’s Stephan Gillich. Computers are saving businesses time and money, but they are not making humans redundant yet. “It is not so much about letting machines make the decisions themselves, as putting more power in the hands of people when decisions need to be made,” says Dr Gavin Michael, global managing director for R&D and alliances at Accenture, the consulting firm.

True artificial intelligence, of the type loved by science-fiction, is still some way out. Computers are smart, but they lack the creative spark of the greatest human thinkers. They are efficient, but as Dr Wesner puts it, they are not yet brilliant.

“It’s possible for a computer to become an impressive chess player, but there is a difference between being impressive and creative,” he says. “A computer is efficient at the number of alternatives it can assess. But that is not the same as brilliance.”

THE ANSWER IS BLOWING IN THE WIND

Danish wind turbine-maker Vestas uses an IBM-designed Firestorm supercomputer to design its turbine blades, but also to monitor power output and weather patterns for wind farms.

The system is the third- largest commercially owned computer in the world, according to Lars Christian Christensen, Vestas’ vice president for plant siting and forecasting.

The company's database holds global meteorological information going back 11 years, with 160 separate parameters including wind direction, atmospheric pressure, cloud cover and rainfall.

The data helps buyers of wind turbines to understand not just how one machine works, but how they function together as a wind farm.

The calculations also help buyers to balance the running costs of a turbine against the electricity it can produce. Siting a turbine in a very windy area might, for example, increase maintenance costs by more than the increase in usable electricity.

“We run the computer at 100% capacity 24/7,” says Christensen, "running tens of thousands of jobs a year. Even the large research institutes cannot deliver the results we demand. It is part of our competitive advantage to have this equipment in-house."

Since 1993, experts have compiled a list of the world’s most powerful computers. The number used in business is growing quickly. Here are some of the fastest.

(NO. 1) K COMPUTER
Operated by the RIKEN Advanced Institute for Computational Science (AICS) in Kobe, Japan, and built by Fujitsu.

(NO. 2) TIANHE-1A
Owned by the Chinese National Supercomputing Center in Tianjin. Based on Intel chips and Nvidia accelerator cards.

(NO. 29) CLUSTER PLATFORM 3000
Developed by HP for an undisclosed French manufactur- ing company.

(NO. 53) FIRESTORM
Operated by wind turbine-maker Vestas, Firestorm consists of 1,222 IBM computers.

(NO. 60) PROLIANT SL390S G7
Built by HP for ENI, the Italian utility company.

(NO. 93) FRONTIER2 BG/L
Operated by French energy company EDF, primarily for nuclear R&D, and built by IBM.