High Performance Computing Systems

Summary of high-performance computing systems at the LCN and Bio-Nano Centre

The LCN has two significant pieces of computational infrastructure:

• A computational grid for the simulation of nanostructures and nanoscale phenomena;
• A visualization grid for the visualization of simulation results and data from experiments.

Both were established as part of a major collaboration between the LCN, the Bio-Nanotechnology Centre (supported by the DTI MNT programme and the London Development Agency) and Sun Microsystems.

The computational grid
The computational grid is centred around a numerically intensive cluster, consisting of nodes with high-performance commodity processors linked by a fast, low-latency network.  This allows  is a distributed-memory architecture with a fast interconnect, based around Sun Microsystems V20z and V40z servers, with dual-core AMD Opteron processors.  The 42 V20z servers each have two dual-core Opteron 275 processors and 8GB shared memory; the seven V40z servers have four dual-core Opteron 880 processors and 16GB shared memory.  The servers are held in two racks and connected by a fast, non-blocking, low-latency Voltaire Infiniband interconnect. 

The V20zs are designed primarily for problems that work on distributed data but require intensive inter-processor communication (problems dominated by Fast Fourier Transforms are a good example).  The V40zs can also be used in this mode, though they are then slightly less efficient because the ratio of communication cards to processors is lower; they are intended primarily, however, for problems that do not parallelize well in a distributed-memory environment (many quantum-chemistry calculations, for instance, fall into this class).   
The total theoretical peak performance of the cluster is 1.01 Tflop (1 Tflop corresponds to 1012 floating-point operations per second); the largest single job the current cluster could support would be a 168-node calculation across all the the V20z servers, using up to 336GB physical memory, at a rate of up to 0.74 Tflop.

The workflow is managed by Sun N1 Grid Engine.  The computational nodes run Red Hat Enterprise Linux. The cluster is attached to fast V490 fileservers and associatated StorEdge 3510 disk arrays with a capacity of approximately 10 TBytes. 

The visualization grid
The high-performance visualization service is provided through an innovative architecture in which the images are first computed on a compute server (a SunFire X4600 with 8 dual-core Opteron 885 processors and 64GB memory) and then passed to dedicated graphics machines (two Sun Ultra 40s with 4GB memory and NVIDIA Quadro FX5500 graphics cards) for 3D mono or active stereo rendering on a large (4m _ 2m) screen located in a dedicated visualization room at the heart of the LCN.  The philosophy underlying this approach is that, with the size of modern scientific datasets, it is more efficient to distribute an image than the underlying data.  The integration between image computation and rendering is managed by open-source Chromium software. 

In order to give maximum flexibility, particularly in scaling up visualization packages from the lab bench, this highest-performance Chromium-based system (running Red Hat Linux) is complemented by a third Ultra 40 with the same hardware specification running MS Windows, to which the screen can be switched dynamically; it is also possible to overlay windows from different sources on the display.  A network of a further five high-performance Ultra 40 graphics workstations throughout the LCN provides high-end desktop graphics for both LCN researchers and BNC clients.  The system is connected to a SunFire X4500 fileserver with a further 24TByte of fast disk.

One benefit of this architecture is that it can make optimum use of the rapid advances in high-performance commodity graphics cards; another is that images can be rendered not only on the dedicated graphics machines but also on other general-purpose machines, either in the LCN or (subject to the availability of sufficient bandwidth) at remote collaborating sites.