jeremi gosney password cracking hpc passwords12

29
Jeremi M Gosney Founder & CEO, Stricture Consulting Group Passwords^12 Security Conference December 3, 2012

Upload: ardit-cankaj

Post on 28-Apr-2015

111 views

Category:

Documents


5 download

TRANSCRIPT

Page 1: Jeremi Gosney Password Cracking HPC Passwords12

Jeremi M Gosney Founder & CEO, Stricture Consulting Group Passwords^12 Security Conference December 3, 2012

Page 2: Jeremi Gosney Password Cracking HPC Passwords12

Password crackers need more power! • Yes, really.

A GPU is great!

More GPUs are better.

How many GPUs are enough?

The Problem

Page 3: Jeremi Gosney Password Cracking HPC Passwords12

Build up • $$ • Motherboard, BIOS, Driver limitations

Build out • Distributing load can be tricky

Potential Solutions

Page 4: Jeremi Gosney Password Cracking HPC Passwords12

Makes remote GPUs appear as if they were local

Implements entire OpenCL 1.1 Standard

Created by Amnon Barak and Amnon Shiloh, Hebrew University

Distributed by MOSIX (www.mosix.org)

Enter Virtual OpenCL (VCL)

Page 5: Jeremi Gosney Password Cracking HPC Passwords12

Free (gratis)

Supports any and all OpenCL devices

Works with any unmodified* OpenCL app

Eliminates the complexity of distributing load

Makes clustering ridiculously easy

VCL – Pros

Page 6: Jeremi Gosney Password Cracking HPC Passwords12

Closed source

Closed license

64-bit Linux only • (Not sure this is really a con) Need highspeed LAN – No Internet clustering

VCL – Cons

Page 7: Jeremi Gosney Password Cracking HPC Passwords12

Provides a pool of shared devices

Completely transparent to app & user

Two-Layer model • Broker node •Executes kernels on compute nodes.

• Compute nodes •Compute.

How VCL Works

Page 8: Jeremi Gosney Password Cracking HPC Passwords12

This is the front-end

• Has your software stack installed

• Only needs the VCL library and broker daemon

• Does not need any OpenCL devices

• Does not need any drivers

The Broker Node

Page 9: Jeremi Gosney Password Cracking HPC Passwords12

This is the back-end

• No software to install outside of VCL back-end daemon

• Needs OpenCL devices

• Needs proprietary drivers & OpenCL runtime

Compute Nodes

Page 10: Jeremi Gosney Password Cracking HPC Passwords12

VCL Architecture

Broker node

Back-end daemon

VCL Library

Broker

Compute node

OpenCL Devices

Back-end daemon

CPU Process

Kernels

Diagrams Courtesy of Amnon Barak

Page 11: Jeremi Gosney Password Cracking HPC Passwords12

Alternate Architecture

Compute node

OpenCL devices

Back-end daemon

VCL Library

Broker

Compute node

OpenCL Devices

Back-end daemon

CPU Process

Kernels

Diagrams Courtesy of Amnon Barak

Page 12: Jeremi Gosney Password Cracking HPC Passwords12

VCL Workflow

Broker node Compute node

Application

Kernel 1 running

Kernel 2 running

Kernel N running

File

System

Diagrams Courtesy of Amnon Barak

Page 13: Jeremi Gosney Password Cracking HPC Passwords12

Network optimization library for VCL

Single call for multiple executions

Direct file I/O to/from OpenCL memory object

Async data transfer with broker

SuperCL

Page 14: Jeremi Gosney Password Cracking HPC Passwords12

SuperCL Workflow

Broker node Compute node

Application Input data from file-system

Kernel 2 running Kernel 1 running

Kernel N running

Output data to file-system

File

System

Diagrams Courtesy of Amnon Barak

Page 15: Jeremi Gosney Password Cracking HPC Passwords12

We can use VCL for password cracking! • You knew this part was coming.

Jens Steube added VCL support for up to 128 AMD GPUs in oclHashcat-plus v0.09

MOSIX were more than happy to help debug and resolve issues

VCL + Hashcat

Page 16: Jeremi Gosney Password Cracking HPC Passwords12

Bandwidth

• Brute force / mask attacks need very little bandwidth

• Wordlist attacks need a lot

Latency

• Slow hashes can tolerate some latency

• Fast hashes cannot

• No SuperCL support in Hashcat

Memory

• Broker node needs a lot of it

• Higher the –n value, the more you need

VCL + Hashcat – Considerations

Page 17: Jeremi Gosney Password Cracking HPC Passwords12

Five 4U servers

25 AMD Radeon GPUs

• 10x HD 7970

• 4x HD 5970 (dual GPU)

• 3x HD 6990 (dual GPU)

• 1x HD 5870

4x SDR Infiniband interconnect

7kW of electricity

Broker daemon runs on a cluster node

Our Cluster

Page 18: Jeremi Gosney Password Cracking HPC Passwords12

epixoip@token:~/oclHashcat-lite-0.11$ LD_LIBRARY_PATH=/usr/lib/vcl vclrun \ ./vclHashcat-lite64.bin -b --benchmark-mode 1 –force oclHashcat-lite v0.11 by atom starting... Password lengths: 1 - 54 Watchdog: Temperature abort trigger disabled Watchdog: Temperature retain trigger disabled Device #1: Tahiti, 2048MB, 1100Mhz, 32MCU Device #2: Tahiti, 2048MB, 1100Mhz, 32MCU Device #3: Cypress, 512MB, 830Mhz, 20MCU Device #4: Tahiti, 2048MB, 1100Mhz, 32MCU Device #5: Cypress, 512MB, 830Mhz, 20MCU Device #6: Tahiti, 2048MB, 1100Mhz, 32MCU Device #7: Cypress, 512MB, 830Mhz, 20MCU Device #8: Tahiti, 2048MB, 1100Mhz, 32MCU Device #9: Cypress, 512MB, 830Mhz, 20MCU Device #10: Cayman, 1024MB, 880Mhz, 24MCU Device #11: Tahiti, 2048MB, 1100Mhz, 32MCU Device #12: Cypress, 512MB, 830Mhz, 20MCU Device #13: Cayman, 1024MB, 880Mhz, 24MCU Device #14: Tahiti, 2048MB, 1100Mhz, 32MCU Device #15: Cypress, 512MB, 830Mhz, 20MCU Device #16: Cayman, 1024MB, 880Mhz, 24MCU Device #17: Tahiti, 2048MB, 1100Mhz, 32MCU Device #18: Cypress, 512MB, 830Mhz, 20MCU Device #19: Cayman, 1024MB, 880Mhz, 24MCU Device #20: Tahiti, 2048MB, 1100Mhz, 32MCU Device #21: Cypress, 512MB, 830Mhz, 20MCU Device #22: Cayman, 1024MB, 880Mhz, 24MCU Device #23: Tahiti, 2048MB, 1100Mhz, 32MCU Device #24: Cypress, 512MB, 830Mhz, 20MCU Device #25: Cayman, 1024MB, 880Mhz, 24MCU [s]tatus [p]ause [r]esume [q]uit =>

Page 19: Jeremi Gosney Password Cracking HPC Passwords12
Page 20: Jeremi Gosney Password Cracking HPC Passwords12
Page 21: Jeremi Gosney Password Cracking HPC Passwords12
Page 22: Jeremi Gosney Password Cracking HPC Passwords12
Page 23: Jeremi Gosney Password Cracking HPC Passwords12
Page 24: Jeremi Gosney Password Cracking HPC Passwords12
Page 25: Jeremi Gosney Password Cracking HPC Passwords12
Page 26: Jeremi Gosney Password Cracking HPC Passwords12

Brute force consistently uses < 8 Mbps

Wordlist attacks on fast hashes use no more than 800 Mbps

Average peak of 88 Mbit per physical card

Ethernet latencies are still an issue • Infiniband helps tremendously

Bandwidth Measurements

Page 27: Jeremi Gosney Password Cracking HPC Passwords12

Benchmarks – Fast Hashes

0 50,000,000,000 100,000,000,000 150,000,000,000 200,000,000,000 250,000,000,000 300,000,000,000 350,000,000,000

NTLM

MD5

SHA1

LM

348 G/s

180 G/s

63 G/s

20 G/s

Page 28: Jeremi Gosney Password Cracking HPC Passwords12

Benchmarks – Slow Hashes

0 10000000 20000000 30000000 40000000 50000000 60000000 70000000 80000000

sha512crypt

bcrypt (05)

md5crypt77 M/s

71 k/s

364 k/s

Page 29: Jeremi Gosney Password Cracking HPC Passwords12

IRC: epixoip on EFnet, Freenode

Twitter: @jmgosney

Keeping in touch