DEEPprojects prototypes
Within the DEEP, DEEP-ER and DEEP-EST research projects various prototypes have been or are being built.
Here is an overview on all systems.
DEEP-EST Prototype System
Download MP4 in HD: [720p] [1080p]
Module | Cluster Module (CM) | Data Analytics Module (DAM) | Extreme Scale Booster (ESB) |
Usage and design target | Applications and code parts requiring high single-thread performance and a modest amount of memory, which typically show moderate scalability. General purpose performance and energy efficiency are essential for the CM. |
Data-intensive analytics and machine learning applications and code parts requiring large memory capacity, data streaming, bit- or small datatype processing. Flexibility, non-volatile memory and different acceleration capabilities are key features of the DAM. |
Compute intensive applications and code parts with regular control and data structures, showing high scalability. Energy efficiency, balanced architecture, packaging and hardware scalability are also important aspects in the ESB design. |
Nodes | 50× with 2 Intel® Xeon® Gold 6146 processors (“Skylake“ generation with 12 cores @3.2 GHz) each | 16× with 2 Intel® Xeon® Platinum 8260M processors (“Cascade Lake“ generation with 24 cores @2.4 GHz) each | 75× one Intel Xeon 4215 Silver processor (“Cascade Lake“ generation with 18 cores @2.5 GHz) each |
Accelerators | - | Per node: 1× NVIDIA® Tesla® V100 GPU 1× Intel Stratix10 FPGA |
Per node: 1× NVIDIA® Tesla® V100 GPU |
Memory | |||
DDR4 | 192 GB | 384 GB 32GB (in FPGA) |
48 GB |
HBM2 | - | 32 GB (in GPU) | 32 GB (in GPU) |
Non-volatile Memory | - | 2 – 3 TB Intel® OptaneTM Datacenter Persistent Memory | - |
Max. Memory BW/node |
256 GB/s | 256 GB/s in CPU 900 GB/s in GPU 77 GB/s in FPGA |
115 GB/S in CPU 900 GB/s in GPU |
Storage | 1x 512 GB NVMe SSD | 2x 1.5 TB NVMe SSD | 1x 512 GB NVMe SSD |
Network | Mellanox InfiniBand EDR (100 Gb/s) with fat tree topology | Switched Ethernet (40 Gb/s), with star topology and EXTOLL TOURMALET (100 Gb/s) with 2D torus topology | EXTOLL Fabri3 (100 Gb/s)* with 3D torus topology |
Cooling | Warm-water | Air | Warm-water |
Max. Power | 25 kW | 25 kW | 36 kW |
Integration | 1× Rack MEGWARE SlideSX-LC ColdCon | 1× Rack MEGWARE | 3× Rack MEGWARE SlideSX-LC ColdCon |
Installation @Jülich | Q1/2019 | Q3/2019 | Q1/2020 |
(*) The initial installation of the ESB will use a Mellanox InfiniBand EDR interconnect, to be upgraded to an EXTOLL Fabri3 solution in Q3/2020.
Cluster Module (CM)
Data Analytics Module (DAM)
Extreme Scale Booster (ESB)
|
|
DEEP-ER Prototype: SDV
Cluster
- 16 dual-socket Intel® Xeon® E5-2680v3 nodes
- Each node: 128 GByte DRAM, 400 GByte NVM
Booster
- 8 Adams Pass Intel Xeon Phi 7210 CPU
- Each node: 16 GByte on-package memory, 96 GByte DRAM, 200 GByte NVM
System
- EXTOLL fabric using TOURMALET NICs with six links of 100Gbit/s each
- Aggregate performance approx. 40 TFlop/s
Storage
- 2 storage servers (spinning disks, 57 TB)
- 1 metadata server (SSDs)
- BeeGFS file system
DEEP Cluster
- 1 Rack with 8 backplanes x 16 nodes (total 128)
- Nodes: 2 x Intel(R) Xeon(R) CPU E5-2680, 32 GB RAM
- Processors: 256 x Intel(R) Xeon(R) CPU E5-2680 (2048 cores)
- Overall peak performance: 45 Teraflops
- Main memory: 4 TB (aggregate)
- Network:
- 1 Gigabit-Ethernet
- Infiniband (QDR)
- 3D torus (FPGA based)
- Power consumption: 50 kW (aggregate)
- Operating system: CentOS 6.3
- Vendor: Eurotech
DEEP Aurora Booster
- 1 rack with 24 (half) backplanes in 12 chassis
- 24 x Booster Interface Cards (BIC)
- 16 x 2 booster nodes (BNs) per chassis (384 total)
- Processor: Intel Xeon Phi 7120X
- Main memory: 6.1 TB (aggregate)
- Overall peak performance: 500 Teraflops
- Network:
- Gigabit-Ethernet
- 3D EXTOLL torus
- Power consumption: max. 150 kW
- Operating system: Linux
- Vendor: Eurotech
DEEP GreenICE Booster
- 1 cube with 32 Booster Nodes (BNs)
- Processor: Intel Xeon Phi 7120D
- Main Memory: 512 GB (aggregate)
- Overall peak perpfrormance: 38.4 Teraflops
- Network: 3D EXTOLL torus
- Power consumption: 10 kW
- Operating system: Linux
- Vendor: UniHD/Megware