Read Dell HPC Solutions Engineering Overview text version

An Open, Standards Based Approach to Building Supercomputers

HPC Advisory Council

March 21-23, 2011 Lugano, Switzerland Reza Rooholamini

Development Approach

Deliver ROBUST, RELIABLE, and SCALABLE solutions that solve customer problems while REMOVING COMPLEXITY and PROVIDING CHOICE...

Infrastructure Software

Messaging Database HPCC

HA

Virtualizatio n

Platform Software / FS

Windows /Linux Operating Systems

ESX/Xen/Hyper-V Hypervisors

OCFS / NFS / GFS

Datacenter Automation, Resource Management/ Orchestration, etc

Server

PowerEdge Servers (Rack / Tower / Blade)

iDRAC, IPMI, BMC, DMC, OpenManage, etc SAS/SATA EqualLogic SANHQ, EMC Navisphere, etc

Fabric

Fibre Channel

Infiniband

FCoE

10GbE/1GbE

Storage

FC SAN

NAS

iSCSI

TAPE

JBOD/RBOD

Global Solutions Engineering

Building Block Approach for HPCC

Applications

Compute Nodes Middleware Parallel Benchmarks (NAS, HINT, Linpack...) and Parallel Applications

MVAPICH Dell MPICH-GM/MX MPICH PowerEdge Servers (Intel andIntel MPI AMD)

Protocol OS

Interconnect Protocol File Systems Interconnect I/O Nodes Platform

Storage Protocol

TCP Linux Ethernet TCP NFS `pNFS

GM/MX

InfiniBand

Windows Myrinet GM/MX Lustre InfiniBand InfiniBand Stornext

Gigabit Ethernet

Myrinet

Dell PowerEdge Servers

InfiniBand

Dell PowerEdge Servers (Intel and AMD)

SCSI and SAS Fibre channel

PowerVault, PS-series and Dell |EMC storage Engineering Global Solutions

Storage

HPCC Program

· Vision: Provide the best price/performance, complete High Performance Computing solution, using standard building blocks and partnering with best of class in HPC. · Proactively analyze HPC technologies, provide best practices to optimize your environments

· Provide integrated products including automation tools for you to design and manage your HPC infrastructure easier

· Evaluate new technologies and provide a window into the future to help you plan for new developments

Global Solutions Engineering

HPCC Components and Enabling Technologies

Lustre NFS/XFS

Platform Lava (Open Source)

Platform LSF (Enterprise) Resource Monitoring / Management Resource dynamic allocation

Checkpoint restarting and Job redistributing Compilers and math library

Application

Benchmark

Job Scheduler Development Tools Middleware / API

Cluster File System Node Monitoring & Management

Dell OMSA (Server Agent) Out-of-Band via iDRAC Automated BMC/BIOS setup

Performance tools - MPI analyzer / profiler - Debugger

- Performance analyzer and optimizer

Operating Systems

Interconnect Protocols

Dell Management Console

Interconnects Hardware Management Hardware

Platform Hardware

Cluster Hardware Software Monitoring & Management

Nagios OM integration GUI/CLI cluster admin tools

Cluster Installation

Global Solutions Engineering

Top 500 Dell Presence ­ November 2010

Rank

40

Site

Lawrence Livermore National Laboratory

System

Dell Xanadu 3 Cluster, Xeon X5660 2.8 Ghz, QLogic InfiniBand QDR

Cores

21756

RMax

166700

Rpeak

243667

45

66

University of Colorado

Lawrence Livermore National Laboratory

PowerEdge C6100 Cluster, Xeon X5660 2.8 Ghz, Infiniband

Dell Xanadu 3 Cluster, Xeon X5660 2.8 Ghz, QLogic InfiniBand QDR

15648

15000

152200

105900

175258

168000

93

107 109

Maui High-Performance Computing Center

Lawrence Livermore National Laboratory NCSA

Dell PowerEdge M610 Cluster, Nehalem -EP 2.8Ghz, Infiniband

Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR PowerEdge 1955, 2.33 GHz, Infiniband

9210

8464 9600

80580

72410 68480

103219

81254 89587.2

110

123

Texas Tech University

KTH - Royal Institute of Technology

Poweredge Cluster, Xeon X5660 2.8Ghz, Infiniband DDR

PowerEdge SC1435 Dual core Opteron 2.2 GHz, Infiniband

7524

9800

68270

61760

84538

86024.4

141

183

NNSA/Sandia National Laboratories

Texas Advanced Computing Center/Univ. of Texas

PowerEdge 1850, 3.6 GHz, Infiniband

PowerEdge 1955, 2.66 GHz, Infiniband

9024

5848

53000

46730

64972.8

62220

222

259 269

Maui High-Performance Computing Center

University of Kentucky R-Systems

PowerEdge 1955, 3.0 GHz, Infiniband

PowerEdge C6100 Cluster, Xeon X5650 2.66 Ghz, Infiniband Dell DCS CS23-SH, QC HT 2.8 GHz, Infiniband

5200

4428 4608

42390

40310 39580

62400

47114 51609.6

349

371

HHMI - Janelia Farm Research Campus

Louisiana Optical Network Initiative

Dell Poweredge R610, Xeon X5550 2.66 Ghz, 10G Ethernet

PowerEdge 1950, 2.33 GHz, Infiniband

4000

5440

35810

34780

42560

50766.1

432

457

Harvard University - FAS Research Computing

Lawrence Livermore National Laboratory

PowerEdge M600 Cluster, Xeon E5410 2.3Ghz, Infiniband

Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR

4096

4032

32410

31860

38175

40320

458

459

Lawrence Livermore National Laboratory

Lawrence Livermore National Laboratory

Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR

Dell Xanadu 3 Cluster, Xeon X5660 2.8 Ghz, QLogic InfiniBand QDR

4032

3480

31860

31790

40320

38976

79

90 287

University of Southern California

Clemson University Saudi Aramco

PowerEdge 1950/SunFire X2200/iDataPlex/IBM dx340 Myrinet 10G

PowerEdge 1950/SunFire X2200/iDataPlex/IBM dx340 Myrinet 10G PowerEdge 1950, Xeon 53xx 3 GHz, Infiniband

13000

11408 4096

94220

85040 38490

119600

104594 49152

22 entries on the latest list

Global Solutions Engineering

Modular HPC Architecture

Raw Data Set Compute Fast Scratch Results

Visualization Archive

Compute

7

Permanent Storage

Global Solutions Engineering

Confidential

Archive

Recommended Host: Dell PowerEdge C6100

Four 2-Socket Nodes in 2U

Intel Westmere-EP 12 DIMMs each 2 GigE (Intel) 1 Daughter Card (PCIe x8) 10GigE QDR IB One PCIe x16 (half-length, half-height) Optional SAS controller (in-place of IB)

Hot Plug, Individually Serviceable System Boards / Nodes Up to 12 x 3.5" drives (3 per node) Up to 24 x 2.5" drives (6 per node)

Each Node:

Chassis Design:

N+1 Power supplies (1100W or 1400W) NVIDIA HIC certified DDR and QDR IB PCIe card certified

Dell High Performance Computing

8

PowerVault MD1200

Versatility and high capacity storage for mainstream applications, with optimal performance in sequential (streaming) applications. · An easy way to expand server capacity and meet versatile business needs with mix drive options · Performance and security for applications that focus on sequential I/O

Key Features

· 2U ­ 12 drive 3.5" enclosure · Supports 2.5" and 3.5" drives · Supports Nearline, 10K and 15K RPM HDD and SAS SSDs · Support for Self-Encrypting Drives · Expands to support 8 enclosures (96 drives)

· Utilizes PERC H800

· Affinity with PowerEdge Servers

· Keep up with environmental demands with 80PLUS® Silver Certified power supplies and improved density.

Great For

· Mainstream and Capacity-Intensive Applications (backup to disk, email, streaming media) · Low cost way to expand the capacity of a single PowerEdge server

Global Solutions Engineering 9

DELL CONFIDENTIAL

Dell PowerEdge C410x:

· 3U chassis (external)

-"Room-and-Board" for PCIe Gen-2 x16 devices - Up to 8 hosts

· Sixteen (16) x16 Gen-2 Devices

Initial Target = GPGPUs Support for any FH/HL or HH/HL device Each slot Double-Wide Individually Serviceable

· N+1 Power (3+1)

- Gold (90%)

· N+1 Cooling (7+1)

Dell High Performance Computing

10

C410x Flexibility: One Host with a Single x16

1:1, 2:1, 3:1, 4:1, (8:1)

1 GPU / x16

Host HI x16 C

PCI Switch

2 GPUs / x16

GPU

Host HI x16 C

PCI Switch

x16

x16 x16

GPU GPU

iPASS cable

C410x

iPASS cable

C410x

3 GPUs / x16

Host HI x16 C

PCI Switch

4 GPUs / x16

GPU

GPU GPU

Host HI x16 C

PCI Switch

x16 x16 x16

x16 x16 x16 x16

GPU GPU GPU

iPASS cable C410x

iPASS cable

C410x

GPU

Dell High Performance Computing

1

C410x Flexibility: One Host with Dual x16

1:1, 2:1, 3:1, 4:1, (8:1) 6 GPUs: 3 GPU / x16

x16 x16 x16 Host HIC

x16

HIC

PCI Switch PCI Switch

8 GPUs: 4 GPU / x16

x16

GPU GPU GPU GPU GPU GPU GPU

GPU GPU GPU GPU GPU GPU

x16

PCI Switch PCI Switch

x16 x16 x16 x16 x16 x16 x16

C410x

HIC

x16

x16 x16 x16

Host HIC x16 iPASS cable

iPASS cable C410x

GPU

Dell High Performance Computing

1

11G - Turbo mode

Energy efficieny gains with Turbo disabled

1.20

Performance/Watt relative to Turbo on (higher is better)

1.18

1.16

1.14

1.12

1.10

- 2% Perf + 21% Power

1.06

1.04

1.02

1.00

ECLIPSE - FOURMILL FLUENT - truck_poly_14m LU - class D WRF - Conus 12km

-5% Perf + 14% Power

Global Solutions Engineering

- 8% Perf + 12% Power

1.08

- 6% Perf +20% Power

11G ­ Optimal BIOS Settings

Optimal BIOS Settings for High Performance Computing with PowerEdge 11G Servers

BIOS Setting Power Management C-State Node Interleaving SMT Turbo Mode DCU/Data Reuse

Energy Efficiency APC Enabled Disabled Disabled Disabled Disabled

Max Performance Max Perf Enabled Disabled Disabled Enabled Enabled

Balanced APC Enabled Disabled Disabled Enabled Enabled

Min. Latency Max Perf Disabled Disabled Disabled Enabled Enabled

Global Solutions Engineering

I/O Bandwidth Comparison

Ohio State University MPI Bandwidth Test

QDR Infiniband (x8 PCIe gen 2) DDR Infiniband (x8 PCIe)

3500 3000 2500 2000 1500 1000 500 0

SDR Infinband (PCI-X)

MB/s

1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072 262144 524288 1048576 2097152 4194304

Message Size (Bytes)

Global Solutions Engineering

GPU:Host Scaling : GPU-HMMER

GPU-HMMER: GPU Scaling

7000 6000 Wall clock (s) 5000

4000

Speedup C410x / C6100 (1) C410x / C6100 (2) C410x / C6100 (4) 1.8X 3.6X 7.2X

3000 2000 1000 0 415 983 1419 Length of HMM 2293

Dell High Performance Computing

16

Resilient Storage ­ Dell NFS Storage Solution (NSS)

Dell HPC NFS Storage Solution (NSS)

· Primary Storage for Small (4 ­ 64 node) cluster · Provide storage for /home directories, applications and long term data storage needs for medium to large clusters

· Small (20TB), Medium (40TB) and Large (80TB) configurations · QDR InfiniBand or 10Gb Ethernet connectivity. · Dell developed best practices and tuning guidance. · Based on XFS File System to provide a single namespace · Built on Dell Power Edge R710 and Dell PowerVault MD1200 · 3 years of Dell PRO Support for IT and Mission Critical 4HR 7x24 onsite pack.

17

Confidential

Global Solutions Engineering

NSS Performance

NSS IPoIB Sequential Writes

1600000 1400000

1200000

Throughput KB/s

1000000

800000

NSS Small

NSS Medium

600000

NSS Large

400000

200000

0 1 2 4 Nodes 8 16 32

18

Confidential

Global Solutions Engineering

An Example of Standard HPCC Bundle

Global Solutions Engineering

What is an Advisor?

· Software application that recommends the best fit Dell products and solutions based on customers specific needs · Available on Dell.com · Example: The HPCC Advisor asks user: ­ OS type?

­ Optimize for performance, power or density

Solution Advisors

Launch

­ Desired sustained or theoretical performance ­ Recommends a solution based on this input. http://dell.com/hpcc

Global Solutions Engineering

Research Opportunities

· File Systems

· Heterogeneous operating system support for parallel file systems · Content addressable storage for unstructured data · Integration of HSM and de-dupe capability with HPC file systems

· HPC in the Cloud

· Features needed in Virtualization layer to support HPCC

· I/O bypass support · Virtualization layer Peer communication capabilities

· Semantic/architecture of an HPCC VM

· Policy-based personalities · Compute node, I/O node, head node as needed

· Unified fabric

· Reliability · Adaptability to application classes

Global Solutions Engineering

GSE Lab in Austin Texas

· HPCC lab with wide array of servers, heterogeneous storage (DAS, NAS, iSCSI, Fibre Channel) and networking products · Similar laboratory environment in Bangalore, India and Limerick, Ireland · Greater than 100 engineers work on HPCC · You have may personal invitation to visit us!

Global Solutions Engineering

Thank You!

· [email protected]

· Dell HPCC Website: www.dell.com/hpcc

­ Optimal BIOS Settings for High-Performance Computing With PowerEdge 11G Servers http://content.dell.com/us/en/enterprise/d/business~solutions~whitepapers~en/Docume nts~HPC_Dell_11G_BIOS_Options.pdf.aspx

­ DellTM | Terascala HPC Storage Solution http://i.dell.com/sites/content/business/solutions/whitepapers/en/Documents/hpc-dellterascala-storage.pdf ­ DELLTM PowerVaultTM MD1200 Performance as a Network File System (NFS) Backend Storage Solution http://i.dell.com/sites/content/business/solutions/whitepapers/en/Documents/hpc-pvmd1200-nfs.pdf

­ Memory Selection Guidelines for High Performance Computing with DellTM PowerEdgeTM 11G Servers http://content.dell.com/us/en/enterprise/d/business~solutions~whitepapers~en/Docume nts~11g-memory-hpc-wp.pdf.aspx · Scalable Storage for High Performance Computing: Red Hat Global File System on Dell EqualLogic - http://www.dell.com/downloads/global/solutions/dell_equal_logic.pdf · NFS Server with 10 Gigabit Ethernet Networks http://www.dell.com/downloads/global/solutions/dell_nfs_server.pdf Solutions Engineering Global

Questions ?

Global Solutions Engineering

Performance gains with Turbo mode

1.10

Performance with Turbo on relative to Turbo off (higher is better) 1.08

1.06

1.04

1.02

IB IB IB

GbE

1.00

GbE

IB

0.98

GbE GbE

0.96 ECLIPSE - FOURMILL FLUENT truck_poly_14m LU - class D WRF - Conus 12km

· Turbo mode enabled helps performance by up to 8%.

16-node PowerEdge M610 cluster with Dual Intel Xeon X5570, 6 x 4GB 1333 MHz RDIMMS

Global Solutions Engineering

Impact of Interconnect on Application Performance

26

Confidential

Global Solutions Engineering

Impact of RAID mode and Drive Configuration on Performance

MD1200 + H800, NearLine SAS, Seq Read

2000

1800 1600

1400

Throughput (MB/s) 1200 R6/6 1000

R5/6

R6/12

800

600 400

200

0

6 Dr

12 Dr

24 Dr

36 Dr

48 Dr

27

Confidential

Global Solutions Engineering

GPU:Host Scaling : LAMMPS Gay-Berne

LAMMPS GB GPU Scaling

1200 1000 Wall Clock(s) 800 600 400

200

Speedup C410x / C6100 (1) C410x / C6100 (2) C410x / C6100 (4) 41X 82X 142X

0 15625 32768 64000 Number of Particles

Dell High Performance Computing

28

Impact of RAID mode and Drive Configuration on Performance

MD1200 + H800, NearLine SAS, Seq Write

1400

1200

1000 Throughput (MB/s)

800 R6/6 R5/6 600

R6/12

400

200

0

6 Dr

12 Dr

24 Dr

36 Dr

48 Dr

29

Confidential

Global Solutions Engineering

Information

Dell HPC Solutions Engineering Overview

29 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

83969