DEEP Projects | Prototypes

Within the DEEP, DEEP-ER and DEEP-EST research projects various prototypes have been or are being built.

Here is an overview on all systems.

Download MP4 in HD: [720p] [1080p]

Module	Cluster Module (CM)	Data Analytics Module (DAM)	Extreme Scale Booster (ESB)
Usage and design target	Applications and code parts requiring high single-thread performance and a modest amount of memory, which typically show moderate scalability. General purpose performance and energy efficiency are essential for the CM.	Data-intensive analytics and machine learning applications and code parts requiring large memory capacity, data streaming, bit- or small datatype processing. Flexibility, non-volatile memory and different acceleration capabilities are key features of the DAM.	Compute intensive applications and code parts with regular control and data structures, showing high scalability. Energy efficiency, balanced architecture, packaging and hardware scalability are also important aspects in the ESB design.
Nodes	50× with 2 Intel® Xeon® Gold 6146 processors (“Skylake“ generation with 12 cores @3.2 GHz) each	16× with 2 Intel® Xeon® Platinum 8260M processors (“Cascade Lake“ generation with 24 cores @2.4 GHz) each	75× one Intel Xeon 4215 Silver processor (“Cascade Lake“ generation with 18 cores @2.5 GHz) each
Accelerators	-	Per node: 1× NVIDIA® Tesla® V100 GPU 1× Intel Stratix10 FPGA	Per node: 1× NVIDIA® Tesla® V100 GPU
Memory
DDR4	192 GB	384 GB 32GB (in FPGA)	48 GB
HBM2	-	32 GB (in GPU)	32 GB (in GPU)
Non-volatile Memory	-	2 – 3 TB Intel® OptaneTM Datacenter Persistent Memory	-
Max. Memory BW/node	256 GB/s	256 GB/s in CPU 900 GB/s in GPU 77 GB/s in FPGA	115 GB/S in CPU 900 GB/s in GPU
Storage	1x 512 GB NVMe SSD	2x 1.5 TB NVMe SSD	1x 512 GB NVMe SSD
Network	Mellanox InfiniBand EDR (100 Gb/s) with fat tree topology	Switched Ethernet (40 Gb/s), with star topology and EXTOLL TOURMALET (100 Gb/s) with 2D torus topology	EXTOLL Fabri3 (100 Gb/s)* with 3D torus topology
Cooling	Warm-water	Air	Warm-water
Max. Power	25 kW	25 kW	36 kW
Integration	1× Rack MEGWARE SlideSX-LC ColdCon	1× Rack MEGWARE	3× Rack MEGWARE SlideSX-LC ColdCon
Installation @Jülich	Q1/2019	Q3/2019	Q1/2020

(*) The initial installation of the ESB will use a Mellanox InfiniBand EDR interconnect, to be upgraded to an EXTOLL Fabri3 solution in Q3/2020.

Cluster Module (CM)

Nodes:
- 50× Intel Xeon 6146 “Skylake“ (2× sockets, 12× cores @3.2 GHz)
Memory:
-   DDR4: 192 GB
-   Max. Memory BW/node: 256 GB/s
-   PCIe NVMe SSD: 512 GB
Network: Fat-tree: InfiniBand EDR (100 Gb/s)
Cooling: warm-water
Max Power: 25 kW
Integration: 1× Rack using MEGWARE SlideSX-LC ColdCon

Data Analytics Module (DAM)

Nodes:
-   16× Intel Xeon 8260M “Cascade Lake“ (2× socket, 24× cores @2.4 GHz)
-   16× Nvidia Tesla V100 GPU
-   16× Intel Stratix10 FPGA
Memory:
-   DDR4: 384 GB
-   HBM2: 32 GB (in GPU)
-   NVM: 2 – 3 TB
-   Max. Memory BW/node: 256 GB/s (with 900 GB/s in GPU)
-   PCIe NVMe SSDs: 3 TB
Network: switched Ethernet (40 Gb/s) and 2D-torus EXTOLL (100 Gb/s)
Cooling: air
Max Power: 25 kW
Integration: 1× Rack using MEGWARE

Extreme Scale Booster (ESB)

Nodes:
- 75× Intel Xeon 4215 “Cascade Lake“ (1× socket, 8× cores @2.5 GHz)
- 75× Nvidia Tesla V100 GPU
Memory:
-   DDR4: 48 GB
-   HBM2: 32 GB (in GPU)
-   Max. Memory BW/node: 900 GB/s in GPU
-   PCIe NVMe SSD: 512 GB
Network: 3D-torus/grid EXTOLL Fabri3 (100 Gb/s)
- Note: in the initial installation phase network will be fat-tree like InfiniBand EDR (100 Gb/s). It will be upgraded to Extoll Fabri3 to reach its final configuration
Cooling: warm-water
Max Power: 36 kW
Integration: 3× Rack using MEGWARE SlideSX-LC ColdCon