Within the DEEP, DEEP-ER and DEEP-EST research projects various prototypes have been or are being built.

Here is an overview on all systems.

Download MP4 in HD: [720p] [1080p]

 

 

Module Cluster Module (CM) Data Analytics Module (DAM) Extreme Scale Booster (ESB)
Usage and design target Applications and code parts requiring high single-thread performance and a modest amount of memory, which typically show moderate scalability.
General purpose performance and energy efficiency are essential for the CM.
Data-intensive analytics and machine learning applications and code parts requiring large memory capacity, data streaming, bit- or small datatype processing.
Flexibility, non-volatile memory and different acceleration capabilities are key features of the DAM.
Compute intensive applications and code parts with regular control and data structures, showing high scalability.
Energy efficiency, balanced architecture, packaging and hardware scalability are also important aspects in the ESB design.
Nodes 50× with 2 Intel® Xeon® Gold 6146 processors (“Skylake“ generation with 12 cores @3.2 GHz) each 16× with 2 Intel® Xeon® Platinum 8260M processors (“Cascade Lake“ generation with 24 cores @2.4 GHz) each 75× one Intel Xeon 4215 Silver processor (“Cascade Lake“ generation with 18 cores @2.5 GHz) each
Accelerators - Per node:
1× NVIDIA® Tesla® V100 GPU
1× Intel Stratix10 FPGA
Per node:
1× NVIDIA® Tesla® V100 GPU
Memory      
DDR4 192 GB 384 GB
32GB (in FPGA)
48 GB
HBM2 - 32 GB (in GPU) 32 GB (in GPU)
Non-volatile Memory - 2 – 3 TB Intel® OptaneTM Datacenter Persistent Memory -

Max. Memory

BW/node

256 GB/s 256 GB/s in CPU
900 GB/s in GPU
77 GB/s in FPGA
115 GB/S in CPU
900 GB/s in GPU
Storage 1x 512 GB NVMe SSD 2x 1.5 TB NVMe SSD 1x 512 GB NVMe SSD
Network Mellanox InfiniBand EDR (100 Gb/s) with fat tree topology Switched Ethernet (40 Gb/s), with star topology and EXTOLL TOURMALET (100 Gb/s) with 2D torus topology EXTOLL Fabri3 (100 Gb/s)* with 3D torus topology
Cooling Warm-water Air Warm-water
Max. Power 25 kW 25 kW 36 kW
Integration 1× Rack MEGWARE SlideSX-LC ColdCon 1× Rack MEGWARE 3× Rack MEGWARE SlideSX-LC ColdCon
Installation @Jülich Q1/2019 Q3/2019 Q1/2020

(*) The initial installation of the ESB will use a Mellanox InfiniBand EDR interconnect, to be upgraded to an EXTOLL Fabri3 solution in Q3/2020. 

 

 

Cluster Module (CM)

  • Nodes:
    -   50× Intel Xeon 6146 “Skylake“ (2× sockets, 12× cores @3.2 GHz)
  • Memory:
    -   DDR4: 192 GB
    -   Max. Memory BW/node: 256 GB/s
    -   PCIe NVMe SSD: 512 GB
  • Network: Fat-tree: InfiniBand EDR (100 Gb/s)
  • Cooling: warm-water
  • Max Power: 25 kW
  • Integration: 1× Rack using MEGWARE SlideSX-LC ColdCon

Data Analytics Module (DAM)

  • Nodes:
    -   16× Intel Xeon 8260M “Cascade Lake“ (2× socket, 24× cores @2.4 GHz)
    -   16× Nvidia Tesla V100 GPU
    -   16× Intel Stratix10 FPGA
  • Memory:
    -   DDR4: 384 GB
    -   HBM2: 32 GB (in GPU)
    -   NVM: 2 – 3 TB
    -   Max. Memory BW/node: 256 GB/s (with 900 GB/s in GPU)
    -   PCIe NVMe SSDs: 3 TB
  • Network: switched Ethernet (40 Gb/s) and 2D-torus EXTOLL (100 Gb/s)
  • Cooling: air
  • Max Power: 25 kW
  • Integration: 1× Rack using MEGWARE

 

Extreme Scale Booster (ESB)

  • Nodes:
    -   75× Intel Xeon 4215 “Cascade Lake“ (1× socket, 8× cores @2.5 GHz)
    -   75× Nvidia Tesla V100 GPU
  • Memory:
    -   DDR4: 48 GB
    -   HBM2: 32 GB (in GPU)
    -   Max. Memory BW/node: 900 GB/s in GPU
    -   PCIe NVMe SSD: 512 GB
  • Network: 3D-torus/grid EXTOLL Fabri3 (100 Gb/s)
    -   Note: in the initial installation phase network will be fat-tree like InfiniBand EDR (100 Gb/s). It will be upgraded to Extoll Fabri3 to reach its final configuration
  • Cooling: warm-water
  • Max Power: 36 kW
  • Integration: 3× Rack using MEGWARE SlideSX-LC ColdCon

 

 

 

DEEP-ER SDV

Cluster

  • 16 dual-socket Intel® Xeon® E5-2680v3 nodes
  • Each node: 128 GByte DRAM, 400 GByte NVM

 

Booster

  • 8 Adams Pass Intel Xeon Phi 7210 CPU
  • Each node: 16 GByte on-package memory, 96 GByte DRAM, 200 GByte NVM

 

System

  • EXTOLL fabric using TOURMALET NICs with six links of 100Gbit/s each
  • Aggregate performance approx. 40 TFlop/s

 

Storage

  • 2 storage servers (spinning disks, 57 TB)
  • 1 metadata server (SSDs)
  • BeeGFS file system

DEEP-ER NAM

2 NAM devices are integrated into the DEEP-ER prototype.

  • FPGA + HMC
  • 2 Extoll links per NAM
  • access via the libNAM library
  • Sequential read/write performance for both links peaks at 11.1 GByte/s and 10.2 GByte/s

 

  • 1 Rack with 8 backplanes x 16 nodes (total 128)                                                         DEEP Cluster
  • Nodes: 2 x Intel(R) Xeon(R) CPU E5-2680, 32 GB RAM
  • Processors: 256 x Intel(R) Xeon(R) CPU E5-2680 (2048 cores)
  • Overall peak performance: 45 Teraflops
  • Main memory: 4 TB (aggregate)
  • Network:
    • 1 Gigabit-Ethernet
    • Infiniband (QDR)
    • 3D torus (FPGA based)
  • Power consumption: 50 kW (aggregate)
  • Operating system: CentOS 6.3
  • Vendor: Eurotech
  • 1 rack with 24 (half) backplanes in 12 chassis                                          DEEP Aurora Booster
  • 24 x Booster Interface Cards (BIC)
  • 16 x 2 booster nodes (BNs) per chassis (384 total)
  • Processor: Intel Xeon Phi 7120X
  • Main memory: 6.1 TB (aggregate)
  • Overall peak performance: 500 Teraflops
  • Network:
    • Gigabit-Ethernet
    • 3D EXTOLL torus
  • Power consumption: max. 150 kW
  • Operating system: Linux
  • Vendor: Eurotech
  • 1 cube with 32 Booster Nodes (BNs)                                          GreenICE Booster
  • Processor: Intel Xeon Phi 7120D
  • Main Memory: 512 GB (aggregate)
  • Overall peak perpfrormance: 38.4 Teraflops
  • Network: 3D EXTOLL torus
  • Power consumption: 10 kW
  • Operating system: Linux
  • Vendor: UniHD/Megware