DEEPprojects prototypes

Within the DEEP, DEEP-ER and DEEP-EST research projects various prototypes have been or are being built.

Here is an overview on all systems.

DEEP-EST Prototype System

Cluster Module (CM)

  • Intel(r) Xeon(r) based HPC Cluster with high single-thread performance and a universal Infiniband interconnect. 

Data Analytics Module (DAM)

  • Intel (r) Xeon(r) based Cluster with non-volatile, byte-addressable memory, one Intel Stratix 10 FPGA and one NVIDIA Tesla V100 card. Nodes will be interconnected by 40 Gb/s Ethernet and a 100 Gb/s EXTOLL fabric.


 

Extreme Scale Booster (ESB)

  • NVIDIA Tesal-base nodes with a small Intel(r) Xeon() CPU and EXTOLL 3D Torus interconnect; objective is to run all applications from the local V100 memeory, and use GPUDirect technology to bypass CPU for network communication.
    The CM will have 50 nodes, and the ESB is planned for 75 nodes - both will use Megware's ColdCon liquid cooling technology. The DAM will have 16 air-cooled nodes with a large memory buildout.

 

Storage

  • A storage and service module will provide high-performance disk storage and  login nodes. This module will use 40 Gb/s Ethernet, and run the BeeGFS parallel file system. 


    A network federation infrastructure ties all the modules together, supporting MPI and IP communication. It is implemented using fabric-to-fabric gateways, and for MPI utilizes high-bandwidth RDMA communication.
    In addition, Network Attached Memory nodes will provide persistent shared memory resources at EXTOLL network speeds, and an experimental Global Communication Engine will drive collective MPI communication on the ESB, with major projected improvements compared to conventional implementations.
  

DEEP-ER Prototype: SDV

DEEP-ER SDV

Cluster

  • 16 dual-socket Intel® Xeon® E5-2680v3 nodes
  • Each node: 128 GByte DRAM, 400 GByte NVM

 

Booster

  • 8 Adams Pass Intel Xeon Phi 7210 CPU
  • Each node: 16 GByte on-package memory, 96 GByte DRAM, 200 GByte NVM

 

System

  • EXTOLL fabric using TOURMALET NICs with six links of 100Gbit/s each
  • Aggregate performance approx. 40 TFlop/s

 

Storage

  • 2 storage servers (spinning disks, 57 TB)
  • 1 metadata server (SSDs)
  • BeeGFS file system

DEEP-ER prototype: NAM

DEEP-ER NAM

2 NAM devices are integrated into the DEEP-ER prototype.

  • FPGA + HMC
  • 2 Extoll links per NAM
  • access via the libNAM library
  • Sequential read/write performance for both links peaks at 11.1 GByte/s and 10.2 GByte/s

 

DEEP Cluster

  • 1 Rack with 8 backplanes x 16 nodes (total 128)                                                         DEEP Cluster
  • Nodes: 2 x Intel(R) Xeon(R) CPU E5-2680, 32 GB RAM
  • Processors: 256 x Intel(R) Xeon(R) CPU E5-2680 (2048 cores)
  • Overall peak performance: 45 Teraflops
  • Main memory: 4 TB (aggregate)
  • Network:
    • 1 Gigabit-Ethernet
    • Infiniband (QDR)
    • 3D torus (FPGA based)
  • Power consumption: 50 kW (aggregate)
  • Operating system: CentOS 6.3
  • Vendor: Eurotech

DEEP Aurora Booster

  • 1 rack with 24 (half) backplanes in 12 chassis                                          DEEP Aurora Booster
  • 24 x Booster Interface Cards (BIC)
  • 16 x 2 booster nodes (BNs) per chassis (384 total)
  • Processor: Intel Xeon Phi 7120X
  • Main memory: 6.1 TB (aggregate)
  • Overall peak performance: 500 Teraflops
  • Network:
    • Gigabit-Ethernet
    • 3D EXTOLL torus
  • Power consumption: max. 150 kW
  • Operating system: Linux
  • Vendor: Eurotech

DEEP GreenICE Booster

  • 1 cube with 32 Booster Nodes (BNs)                                          GreenICE Booster
  • Processor: Intel Xeon Phi 7120D
  • Main Memory: 512 GB (aggregate)
  • Overall peak perpfrormance: 38.4 Teraflops
  • Network: 3D EXTOLL torus
  • Power consumption: 10 kW
  • Operating system: Linux
  • Vendor: UniHD/Megware