Billion dollar brains

Once known only as ‘Site Y’, Los Alamos National Laboratory in New Mexico was first established as part of the Manhattan Project as the Allies began secret development of the world’s first nuclear weapon.

IBM’s Cell blade

Fifty years on, one of the laboratory’s primary purposes is to run the US nuclear stockpile stewardship programme. While this demanding task no longer involves nuclear testing, it requires a small army of engineers and researchers who run tests on the US’s ageing arsenal of nuclear weapons.

One of the key elements of this programme is the running of complex simulations on room-sized supercomputers to understand the dynamic properties of ageing nuclear materials and model how the weapon’s various components might fare over time. The somewhat more sinister flip side of this programme is ensuring that as well as being safe, the weapons are also ready for use.

An essential part of this task is the planned construction of the world’s most powerful supercomputer, named Roadrunner, the next-generation supercomputer that will help keep the nuclear weapon stockpile of the world’s most powerful nation safe.

Part-funded by the US Department of Energy’s National Nuclear Security Administration, the machine is to be built by IBM in conjunction with Los Alamos. It is estimated that, once completed, Roadrunner will be the first supercomputer to break the mythical barrier of the ‘petaflops.’ Flops (floating point operations per second) is a measure of computational power and refers to the number of calculations that the system is capable of performing.

Until Roadrunner comes online, the most powerful computer available was also built by IBM: Blue Gene L at the Lawrence Livermore Laboratory in California. This was the machine that led the charge in a new generation of powerful computers, easily eclipsing what was then the world’s most powerful, Japan’s Earth Simulator. While Blue Gene is almost an order of magnitude more powerful than Earth Simulator, Roadrunner is set to be up to five times more powerful again.

Over the past few years the Blue Gene system has continually beaten all records in computational power, peaking at a massive 207 teraflops. A teraflop is one billion operations per second, whereas a petaflop — the holy grail of scientific computing — equals 1,000 billion operations a second. It is estimated that Roadrunner will be able to reach a hitherto unthinkable 1.6 petaflops.

According to David Jursik, IBM’s vice-president of Deep Computing, even though Roadrunner will have the crucial job of managing the nuclear stockpile there will also be a huge amount of potential applications in the areas of general science, particularly for number-crunching in big physics and chemistry experiments. ‘This will be much bigger than anything that has been built before,’ he said. ‘Roadrunner really steps it up again to the next level from Blue Gene.’

Los Alamos spokesman Kevin Roark explained that Roadrunner’s principal task will be verifying weapons codes, and looking at very detailed computer models of hydrodynamic systems to analyse what specifically happens when a nuclear weapon detonates. Roark said that current limitations in computing power mean that such simulations typically take months and consequently aren’t performed that often. With Roadrunner, he claimed, it will be possible to carry out 3D simulations of a nuclear weapon detonation in just a few days. This should make it possible to subject the US nuclear arsenal to far more rigorous and frequent analysis than is currently possible.

Not all of the machine’s time will be spent on nuclear weapons. It will also be used for a wide variety of theoretical and computational science experiments, where its considerable powers will be brought to bear on challenges such as modelling climate systems and simulating exotic materials at the atomic scale.
It will not, however, be available for outside use, in the way that many in European research facilities are. ‘Roadrunner is a behind-the-fence facility, not for use outside the weapons complex,’ stressed Roark.

According to IBM’s Jursik, one of the limiting factors when building supercomputers is the fact that a natural limit in chip manufacture has been reached in terms of how compact they can be made.

‘In the past 20 or 30 years to make it more powerful you just made smaller chips more powerful but now chips are as small as they can get,’ he said. ‘So people are simply adding more CPUs to their chips and looking at different ways of maximising the power. Roadrunner is one answer, but there will be others.’

The innovation behind Roadrunner lies in its hybrid nature. Alongside the more conventional AMD Opteron processing chips, the supercomputer will also be fitted with an equal number of Cell chips, which were originally designed for Sony’s Playstation 3 games console. While the approach for Blue Gene was to use massive amounts of low speed, low-power chips, the design for Roadrunner is reliant on the incredible acceleration provided by the Cell chips. These were specially designed to carry massive amounts of data at high speeds and have found their way into a number of applications beyond the gaming market since their inception (The Engineer, 3 July).

When completed, Roadrunner will cover an area of 12,000 sq ft and comprise around 16,000 AMD Opteron processors and another 16,000 Cell processors. Despite the fact that the machine is to operate using standard off-the-shelf hardware, its hybrid nature will demand extremely advanced software to orchestrate the two banks of processors.

The system will be fitted with intelligent ‘Hybrid Programming’ software, which can deploy tasks to the different parts of the machine depending on their specific requirements. For example, ‘standard’ computational tasks and processes involving communication and memory within the machine will be run via the Opteron processors. However, more complex and highly intensive or specialised calculations will be sent to the Cell processors to deal with. ‘We are hoping that this will define a set of programming tools that will be useful for the rest of the computing community,’ said Jursik.

While IBM is providing the processors and nodes for Roadrunner, the responsibility for the infrastructure will be shouldered by Los Alamos. Casual observers might expect the cooling and power demands of such supercomputers to throw up some significant engineering challenges, but according to Roark running Roadrunner is all in a day’s work for the hi-tech facility. ‘We have chillers capable of cooling components that run extraordinarily hot — our computing floor isn’t even near capacity — we have room, cooling and power consumption to spare — none of those are issues.’ he boasted.

One of the most interesting aspects of this infrastructure, however, is the chilling system. Currently used to cool the Q-massively powerful supercomputer and two major Linux clusters known as Lightning and Bolt, this system forces cold air up through a honeycomb flooring system. As the chilled air is warmed by the computers it rises, is pulled out of the computing floor by fans in the walls and eventually vented back into the chilling system.

But while engineers prepare to begin Roadrunner’s installation, rival developments are gathering pace that may cut short its supremacy. ‘There are already plans to build a petascale machine in the EU, while the Japanese have early plans to build a system with a power of 10 petaflops,’ said Jursik.

In fact, The Engineer’s research indicates that the most advanced of these rival efforts is the Japanese project, where, with 110bn Yen (£497m) of government funding, Japanese research agency Riken hopes to have a 10-petaflop computer by 2012.

According to Dr Ryutaro Himeno, development group director at Riken’s Next-Generation Supercomputer R&D Centre, the powerful machine will be used to model interatomic forces at the heart of reactions between proteins and other molecules. Himeno explained that a 10-petaflop machine will be able to deal with one million atoms, which, over a calculation time of 24 hours, will make it possible to simulate protein reactions for 20 nanoseconds. He added that the machine will also stimulate progress in other areas such as geoscience, environmental science, nanoscience, and manufacturing.

Indeed, Riken’s intention is that, unlike Roadrunner, its machine will be made available to industry. According to a Riken spokesperson, the system could, for instance, be used by the engineering industry to improve aerodynamic designs through fluid flow simulations. Another real-world benefit could come in materials science. According to the spokesperson, the computer could be used to test all of the synthetic ratios of every possible material to discover new materials for fuel cells.

Despite this, the Japanese group is already considering the limitations of a computer with once-unthinkable processing capability. Himeno claimed that a 10-petaflop machine will be too slow to model the tissue, blood flow and movement of an entire human body and said that his group hopes to ultimately develop exaflop machines (one quintillion floating point operations per second), which would be able to ‘simulate the phenomenon of life itself’.

Closer to home, plans for a European supercomputer that breaks the petaflop barrier are somewhat murkier. According to Hugh Pilcher-Clayton, head of high end computing at the EPSRC, plans for such a machine are embryonic, with European researchers and scientists currently developing a scientific case.
According to a scientific roadmap published two weeks ago by the European Strategy Forum on Research Infrastructures (ESFRI), there is possibly a need for a powerful supercomputer at the pinnacle of a pyramid of smaller facilities.

But Prof John Wood, chief executive of Central Laboratory of the Research Councils at Oxfordshire’s Rutherford-Appleton Laboratory and chairman of ESFRI, suggested that hugely powerful supercomputers are not necessarily exactly what is required by the European research community. ‘There is an active debate on whether one big general machine is required or several machines focused to deliver in specific areas — we do have the feeling that countries are falling over themselves to have the latest most powerful machines without regard to whether there is sufficient user capability,’ he said.
Meanwhile, work is gathering momentum on the UK’s next ultra-fast number cruncher. While it won’t come near the processing speeds promised by Roadrunner, the High End Computing Tetrascale Resource (HECTOR), which will be switched on next April, will be able to run at speeds of up to 100 teraflops. This will make it faster than anything currently available in Europe. However, Jennifer Houghton, project manager for Hector at the EPSRC, said that a location for the machine has not yet been decided.

Based on a Cray XT3 supercomputer, Hector will be available for the entire UK research community to use for experiments including environmental modelling, atomic physics experiments, as well as computational engineering and materials simulation.

The facility is expected, for instance, to be particularly useful in the planning and research of experiments carried out at large scale facilities such as the Diamond Synchrotron in Oxfordshire, as well as at the envisioned 4GLS project (The Engineer, 18 September).

Core engineering applications are likely to include improving the understanding of turbulence, a phenomenon that is exceptionally difficult to predict and analyse mathematically. In one proposed project a group at Glasgow University hopes to use the machine to enhance understanding of the unsteady flow around helicopter rotor blades. The facility could also yield breakthroughs in materials development.

For instance, according to the scientific case for Hector, most crack simulations are still only two dimensional, but with increased computer power these simulations could become fully 3D and enable engineers to address issues such as the structure of steps on the crack edge and their effect on a range of materials’ properties such as resistance to chemical attack.

Back in New Mexico, the first components of Roadrunner have started arriving. Over the next couple of years, IBM will begin installing the first sets of hardware for the supercomputer. Work will then begin in earnest on the programming systems that will divide the work between the two sets of processors and begin the quest to break the petaflop barrier.

But, however powerful it is, things move quickly in the world of supercomputers and Roadrunner may not last for long as the world’s most powerful supercomputer.