In general, when ExtremeTech or another technology website discusses supercomputers, it’s always in terms of speed; teraflops, petaflops, the rush to exascale, and so on. There’s a reason for this, of course: After decades of indoctrination by Intel, processing speed is something that nearly all of us can relate to.
What’s rarely discussed, however, is thepurpose of supercomputers. Really, theworld’s fastest supercomputer recently hit a peak of 10 petaflops — 10 quadrillion calculations per second, or around 200,000 times faster than your Core i7 Sandy Bridge computer — but what does it do with all of that power?
To answer that question, we first need to look at the architecture of supercomputers.
Supercomputers are, for the most part, nothing like your PC. In some cases supercomputers are built with x86 CPUs from AMD and Intel, or GPUs from Nvidia or AMD, but that’s where the similarity ends.
Supercomputers normally make use of customized compute units (called blades) which usually house multiple nodes (CPUs, GPUs). In the case of the Cray XK6, the most powerful blade in the world, each blade contains four nodes, and each node houses a 16-core AMD Opteron CPU and Nvidia Tesla GPU, and 16 or 32GB of RAM. These nodes are connected together with a proprietary interconnect (usually optical). Multiple blades are then stacked together in racks (again, optically networked), allowing for tens of thousands of nodes to be crammed into a large room.
A little known fact: Supercomputers are water-cooled, which not only saves money (hot CPUs leak more power) but also allows them to run faster. In the case of K, the world’s fastest supercomputer, there are some 88,128 compute nodes — 88,128 8-core SPARC64 VIIIfx CPUs — and each one is connected up to what is probably the most complex water cooling system in the world.
Finally, you need some seriously awesome software to control the beast, which nowadays is nearly always a customized version of Linux. Each supercomputer manufacturer (IBM, Cray, Fujitsu) usually starts with a Linux distro of choice, and then makes significant changes to tailor the OS to the specific hardware. It is the operating system’s job to minimize the time each node spends waiting for new data to arrive. This is involves a very intricate job of task scheduling and memory management. Don’t forget, a supercomputer generally has thousands of gigabytes of RAM, and sometimes hard drive storage in the petabyte range.
The end result is a supercomputer that has tens of thousands of nodes that (hopefully) act in parallel. If you imagine a single, futuristic Intel CPU that has ten thousand cores in the same package, that’s the idealistic end-goal that supercomputer designers are aiming for.
The cost of ownership, by the way, is in the hundreds of millions range. Not only do you have the installation cost, but a supercomputer uses megawatts of power, too. Jaguar, the third fastest supercomputer in the world (pictured above), cost $104 million to install and uses 7 megawatts — and every megawatt, at $0.10/kWh, equates to roughly $1 million per year. Add in the dozen or so technicians that you need to keep a supercomputer running, and you’re looking at an annual cost of $10 million or more.
But what the heck can you do with 88,128 parallel processors?
The short answer is: Not much. Or at least, nothing that mere mortals like you or I usually concern ourselves with. Because of the Linux OS, you wouldn’t even be able to play Crysis 2 on a supercomputer — sheesh!
For the most part, supercomputing (sometimes referred to as high power computing or HPC) is used to model simulations or perform brute force calculations. Supercomputers are research tools. Historically (in the ’70s) this was weather forecasting and aerodynamics (planes, space craft, cars), then nuclear weapons simulations and radiation shielding modeling (the height of the Cold War in the ’80s), and in recent years the emphasis has been on cracking decryption and molecular dynamics modeling.
Let’s break down a few of these applications so you can see why they’re run on supercomputers.
With weather forecasting, it is the sheer amount of data that needs to be processed. There are hundreds of thousands of weather stations around the world, each one recording dozens of variables (wind speed/direction, temperature, humidity, pollen count). All of this data needs to be compared to last night’s data, last week’s data, and historical data, all of which must be fetched from huge memory and storage banks. Generally, most of these calculations can occur at the same time, and so massive parallelism means that weather forecasts can be produced in seconds or minutes, rather than hours.
With nuclear weapons simulation, scientists model every step that a nuclear bomb takes from storage in a silo to explosion. We’re talking about a level of simulation that models whether the heat and vibration from a transport truck can affect a nuclear missile’s control systems — but also the simulation of what actually happens when a nuke explodes. Is it better to explode it at higher altitude? What if we throw in a bit more plutonium? And so on. Again, it’s the sheer number of variables that makes massive parallelism such a boon here. One processor can work on the effects of humidity, while another can work on the fluid dynamics of fissile uranium.
Brute force decryption has fallen out of vogue in the last few years (it simply takes too long to break 256- or 512-bit encryption ciphers), but in the ’90s you could use a supercomputer to break 56-bit DES encryption in a few minutes or hours. Here, massive parallelism simply lets you try different passwords at the same time; so if a single CPU can try a thousand passwords per second, a supercomputer with 10,000 CPUs can try 10 million passwords per second. In general, though, brute force attacks take too long to be worth it — though that isn’t to say that intelligence agencies around the world don’t try (most supercomputers are state-owned, incidentally).
Finally we move on to by far the most exciting use of supercomputers (and the main reason for building new supercomputers): Molecular dynamics modeling. Basically, molecular dynamics is the simulation and analysis of the movements of atoms and molecules, and as such it covers the physical and mechanical aspects of almost everything in the known universe. To put this into perspective, Tianhe-1A, the second most powerful supercomputer in the world, ran a simulation involving 110 billion atoms through 500,000 time steps. In every one of these steps, Tianhe-1A has to analyze the relationships between each and every atom. These calculations took three hours to complete and accounted for 0.116 nanoseconds of simulated time — and this is on a computer capable of processing two quadrillion calculations per second.
The most prominent example of molecular dynamics modeling Folding@Home, a distributed computing project that tries to work out the processes behind protein folding (incorrect protein folding is believed to cause a range of diseases, including Alzheimer’s and cancers). Molecular dynamics is also used in pharmaceutical research — and indeed a lot of the world’s largest, privately-owned supercomputers are owned by Big Pharma. There has been a huge surge in molecular dynamics modeling since the completion of the first human genome project in 2000, and the many iterations since. In this area, supercomputers analyze how genes interact with one another, the effects of changing or deleting a gene, and the effect of gene therapy.
Molecular dynamics is also used to model various aspects of astronomy, including the first few seconds after the Big Bang (it’s hard to model every atom in the universe compressed into a tiny point!) and what happens when a star goes supernova. Molecular dynamics brings new levels of understanding to just about every chemistry or physics experiment, so it’s not hard to see why everyone is rushing to build peta- and exascale supercomputers.