The HUYGENS supercomputer system description
The final configuration of the new National Supercomputer, when
fully operational in mid 2008, will be based on IBM's next
generation of POWER systems. The system will consist of a
significant number of so-called Compute Nodes (CNs). These CNs are
connected to each other through an InfiniBand (IB) network,
enabling efficient execution of MPI-based applications which run
across more than one CN. With respect to memory, the final
configuration will meet the NCF procurement requirements of 4 - 8
GB per processor core.
The ratio between the standard CNs and the memory CNs in the
system will be about 4 to 1. Furthermore, 2 standard CNs serve as
front-end nodes for interactive work and submitting jobs to the
batch system. Also, 2 additional standard CNs are being used for
testing purposes (new software releases, etc.).
Lots of information on the next generation POWER technology is still
confidential and will be released with the IBM announcement. Hence, the
details mentioned of the expected next generation POWER design are based
upon an article from RealWorld (view
article), which do not necessarily
reflect IBM's planned announcement. This article contains general
information about a POWER6 chip architecture which is used in different
configurations across a broad range of commercial and technical
The processor core of each CN will be based on IBM's next
generation of POWER processors, like POWER5 able to perform 4
floating-point operations per clock cycle on 64-bit numbers. The
next generation Power processor is expected to run at a clock
frequency between 4.0 and 5.0 GHz. The total peak performance of
the HUYGENS system will be more than 60 Tflop/s.
Huygens will be using the newest processor design within IBM's
family of POWER processors, which exist already for 17 years with
the introduction of the first RS/6000 in 1990.
It is based on RISC technology (Reduced Instruction Set
Computer), which enables simple instructions to be executed in a
pipelined fashion. Memory hierarchy is set up with various levels
of caches between the processor registers and main memory. From
main memory, the route of data to the processor registers is as
follows: L3 cache amounts to 32 Mbytes, shared between two cores,
to L2 cache of 4 MBytes per core, and finally to L1 cache of 64
kBytes per core.
An essential part of the overall system configuration is the
interconnect between the CNs. It is based on InfiniBand technology,
delivered by QLogic switches. Based on IB4x technology, each CN has
links to the IB network. IB4x 288-port IB switches will be used,
which means that each CN can be connected to each IB switch.
IB technology has been chosen because of their advantageous
latency specifications, which is important for the performance of
MPI applications running across CNs. The MPI latency is in the
order of a few microseconds.
Storage and IO Bandwidth
NCF/SARA's procurement has turned out to be very challenging
with respect to IO bandwidth. This has been done on purpose, since
the application workload on the system has a relative large amount
of IO intensive codes. For this reason, the storage environment had
been designed very carefully. First, there is a large amount of
system-wide accessible storage, which is typically used for users'
home file systems. This will amount to ca. 270 TB. Using IBM's
Global Parallel File System (GPFS), each CN has access to the full
range of home file systems. The home file systems can be viewed as
relatively far away from the CNs, although their aggregate IO
bandwidth is approximately 12 GB/s.
In addition to the home file systems, there are two types of
scratch file systems. These are fast scratch file systems and
shared scratch file systems.
Fast scratch file systems are file systems that are local to a
CN, and are typically meant for jobs which run within one CN, or
for jobs which do not have shared IO requirements. Each CN has such
a fast scratch file system. The total aggregate performance is
about 24GB/s across all standard CNs, with an additional 22 GB/s
across high memory nodes. The amount of fast scratch storage
aggregates to ca. 300 TB for all CNs together.
Shared scratch storage is an additional level of fast storage
which is available to all CNs. It is typically meant for jobs which
run across several CNs and for which each MPI process needs access
to shared data. The amount of shared scratch storage is
approximately 130 TB.
The total amount of storage is 270+300+130 = 700 TB usable
storage. All storage is FC-based. A Storage Area Network based on
Brocade switches will be used to grant access to the data from the
Archiving and Backup
As explained above, the system comes with a large amount of
storage for home file systems. Until now, home file systems on the
SGI system were run with automatic migration software (DMF) to
tape, so that file systems never filled up. Drawback was that large
files were moved to tape rather quickly, while retrieving data from
tape could be rather cumbersome.
For the new system, SARA expects to run a quota strategy on the
home file systems. Behind the home file systems, an archive file
system will be run, which is governed by DMF to interact with tape
storage. Users will have to move their files from the home file
system to the archive file system explicitly. Once on the archive
file systems, files may be moved to tape automatically.
Backup is done using IBM's TSM software, which is transparent to
In their Request-for-Proposal, NCF/SARA has expressed its
preference for a Linux-based Operating System. IBM will run SUSE Linux on the system, with its
Fortran, C and
C++ compiler technology. Performance analysis tools are
available to the users of the system. Numerical libraries are
Parallel ESSL (P-ESSL). IBM's Parallel
Environment (PE) will be used for preparing and running of MPI
The scheduling software is an important part of the system.
Users logon on one of the two interactive CNs, and perform their
compilations and small testing on these front-end CNs. Once they
are ready for a larger parallel job, the batch system comes into
action. Depending on the actual requirements of the job (how many
processor cores, how much memory needed, large IO requirements,
etc.), the batch system decides where the job will run on the
system. Clearly, memory-demanding jobs will need to run on the
memory CNs, etc. The actual scheduler is IBM's
Another important piece of software is the global file system
software. As described in the storage part, the home file systems
and the shared scratch file systems rely on a global file system.
IBM's Global Parallel File System (
GPFS) will be used for this.
- 24-inch wide compute racks to hold the CNs
- 19-inch wide storage racks to hold the storage and SAN
- 19-inch wide racks to hold the InfiniBand switches
- 19-inch wide racks to hold networking equipment
Support and Services
The system will be delivered with both hardware and software
support during extended office hours. Furthermore, IBM has
committed to a large number of support days to both system
administrators at SARA and to users in the field. This includes
scripting, accounting, but also analysis and
optimization/parallelization of user applications, and courses and
lectures to the user community.
IBM deliveries begin in June 2007 with an initial configuration,
which is significantly more powerful than the current
configuration. This serves as a platform for the final system,
which is expected to be available to users by August 2008.
The initial configuration will be delivered to SARA in June
2007, and is expected to be operational for users by August 2007.
It will be based on IBM's System p575 with Power5+ processors,
running at 1.9 GHz clock frequency. The total peak performance will
be 14.6 Tflop/s, with CNs with InfiniBand switches connecting them.
The software stack is identical to what will be available on the
"next generation" Power-based system.