Notes on GPU-based Hash Computation

2011-10-29 16:03:45 by chort

In the last few weeks I've learned a lot about applying GPUs to break password hashes. I'd like to thank @ErrataRob for writing the blog post that got me started in this field. If you haven't read Rob's post, I highly recommend you do that first, because this post builds on it. Don't buy a graphics card until you've read my post though, because there are some important updates.

This post is written with the assumption that you'll be using the oclHashcat family of programs and utilities. The first question I had was "which is better, ATI or nVidia?" You can read the arguments here. Essentially ATI has better price/performance ratio for cracking a small number of hashes concurrently, but nVidia does a better job with very large hash lists and has better driver support. Next I was curious whether any performance gap existed between Windows and Linux, which this post settles. When setting up your machine, you'll need to be aware of what supporting SDKs and/or drivers you'll need. Pay attention to the last section of this page for the current platform recommendations. Keep in mind these guidelines probably apply to any software implemented with OpenCL, not just oclHashcat.

I decided to buy an ATI card, because of the lower cost of acquisition. Since there's no performance incentive to run Windows, I decided to go with Linux. I didn't have a favorite Linux distribution (because they all suck, from my experience) so I decided to see what most oclHashcat users run. From the benchmarks on the website it looked like the author (atom) used Ubuntu. He confirms it in several forum posts, including this one. That settled it for me. If it's developed and tested on Ubuntu, that is what I'll use. I went with the same point release (10.10) and architecture (64bit). I was pleasantly surprised by how little frustration this flavor of Linux causes (mind you there's still frustration, just less than I've had with other Linux projects). Note: atom has since upgraded to Ubuntu 11.04 and so have I; however, Ubuntu 11.10 does not work with Catalyst 11.4/SDK 2.4 as of this writing (October 29, 2011).

One thing I haven't addressed yet, the aspect that has the largest impact on performance, is which card to buy. I was so excited to get going that I didn't do much research prior to purchasing a card. On a suggestion I went with a Radeon HD 6950 that came slightly overclocked from the manufacturer. It does have fairly good performance, but it costs roughly $250 to buy new (with taxes, etc) and does generate a fair amount of heat. After I looked around a bit more I was informed that the general method of determining GPU performance for hashing is multiplying shaders by shader clock frequency. I also came across a table that shows reference design shaders & clock speed for many graphics cards. I'm not sure how accurate that is for nVidia cards, but it seems to represent reality for AMD/ATI cards fairly well. The interesting thing to note is that a Radeon HD 5870 out-performs every card other than a 5970 and 6990.

I've discovered that it's impossible to find Radeon HD 5970s anywhere, and that Radeon HD 6990s suffer from heat and power issues (too much of both). On a whim I decided to search Craigslist for 5870s and, to my delight, there were several for sale. Not only were there used 5870s for sale, they were also cheaper than new 6950s. That's right, you can get a faster (and cooler) card for less. The catch here is that I live in the San Francisco Bay area (best place on Earth!), so there's a good secondary market that doesn't exist in most other metro areas. I have been told there are a lot of 5870s for sale on eBay, but I don't have the stomach for that circus. My recommendation if you're in the market for a hash-cracking GPU card is to find the local LAN-gaming groups. One of the sellers I talked with through Craigslist was a member of such a group and he relayed that all the members of his group were starting to sell their HD 5870s to buy newer cards (mostly 6970s). Reach out on Twitter or Craigslist to let people know you're interested in buying a Radeon HD 5870 (or 5970!) for cash. You might be surprised. If you're in an online game clan/guild, get the message out on their forum or group chat.

A final, but important note on hardware is the use of VGA dummy plugs. If you're using Linux (or for some reason an old version of the Catalyst drivers on Windows) and you have multiple GPUs, you may* need to make dummy plugs for all the graphics cards that don't have a monitor attached to them. In my case I bought an AMD A8-3850 CPU that came with a built-in Radeon HD 6550D GPU. That means my machine has two graphics cards, but only one had a monitor attached, so I had to make a dummy plug. While this was initially a shock, the components were cheap (you can get resistors at Fry's if Radio Shack doesn't have them) and they're so easy to put together that my kid made several of them.

Now we're down to the nitty-gritty of installing the software, which is the most nettlesome part of the process. Recall I'm using Ubuntu, so the process may be different for you. To begin, it's very important that you use the driver and SDK versions specified here. At the time of this writing (October 29, 2011) the Catalyst driver version to use for Linux is 11.4 and the SDK version is 2.4. This is likely a moving target, so check the Hashcat Wiki for a definitive answer.

Make sure when you download the currently recommended drivers from the official driver page. Prior to installing the AMD/ATI drivers, you'll need to remove the existing drivers that shipped with your OS. For Ubuntu 11.04 (Natty) you can follow the instructions here. Next you'll want to build and install the official AMD/ATI Catalyst drivers. You should be able to follow the instructions here.

Next you need to install the AMD APP SDK (if using a Radeon GPU on Linux--don't do this for Windows!). The steps are roughly as follows (lifted from this post on a Bitcoin forum).

# 32bit
$ wget

# 64bit
$ wget

$ sudo tar xvfz AMD-APP-SDK-v2.4-lnx??.tgz -C /opt
$ sudo tar xvfz /opt/AMD-APP-SDK-v2.4-lnx??/icd-registration.tgz -C /
$ echo export DISPLAY=:0 >> ~/.bashrc

# 32-bit
$ sudo sh -c 'echo "/opt/AMD-APP-SDK-v2.4-lnx32/lib/x86/" >> /etc/'

# 64-bit
$ sudo sh -c 'echo "/opt/AMD-APP-SDK-v2.4-lnx64/lib/x86_64/" >> /etc/'

$ sudo ldconfig

If you're using multiple GPUs you'll need to use an Xorg.conf that supports multiple displays, otherwise OpenCL will only see the GPU that's mapped to a display. I was able to use the AMD control panel in Gnome to set a second desktop. You can also manually setup your Xorg.conf by duplicating the existing adapter, screen, and monitor sections that were created by aticonfig and simply incrementing the minor number by one (simple example, blah-[0]-[0] becomes blah-[0]-[1], etc). You will probably need to use lspci and your Xorg logs to figure out the PCI IDs of the remaining GPUs. I spent a significant amount of time monkeying with that, so do your research ahead of time. Sorry I don't have direct references--it seems I closed those tabs prematurely before writing this post. Also, if you're using a motherboard or APU with a built-in GPU that you want to use for cracking, be sure to force multi-display mode in your BIOS. Mine automatically disabled the on-board GPU when a dedicated graphics card was present.

If you're trying to use Vino (built-in VNC with Ubuntu) for remote desktop access, it's going to throw fits with multiple displays. I followed the hack here to get it working. Note that you need to add that line just prior to the end of your /etc/gdm/Init/Default file. There are some guides for getting Vino working for the root user, so you can use Vino to login to display the gdm login dialog (by default Vino only works after you have a user logged in with an Xsession). Make sure you SET A VNC PASSWORD for both your own users and root. It would be really stupid if someone owned your box by making an unauthenticated VNC connection while your desktop had rootshells open.

You should now be ready to download and install the various oclHashcat tools from (or any other OpenCL tool) and get cracking!

There are lots of advanced aspects to computing hashes with oclHashcat, such as the application of masks, which wordlists to use, how to optimize them, what commandline flags to apply, etc. I'll leave those for future posts. Also, there's a lot of great information already written on those topics. In fact, one of the things that makes the complicated procedure of installing the correct drivers and SDKs tolerable is that Bitcoin miners have the same pre-requisite requirements, so they've written a great deal of the documentation and troubleshooting material. I'll leave you with a list of other references that have been useful to me:

The Unofficial Wiki for the AMD Linux Driver
Ubuntu Forums and Ubuntu Forums Archive
Modeline Calculator (for Xorg.conf)
Hashcat Wiki
Skullsecurity password lists
Cyberwar Zone (DRINK!) password lists
Irongeek Password Exploitation Class
TobTu Tools
Ivan Golubev's GPU Speed Estimations

* There are conflicting reports on whether dummy plugs are required under Linux. I wasn't able to use all my GPUs without using dummy plugs, and most of the photos I've seen of multi-card cracking machines have had dummy plugs, but some people have stated they're using multiple GPUs under Linux with the Catalyst 11.4 drivers without needing dummy plugs. YMMV.


at 2011-10-31 06:25:39, atom wrote in to say...

nice writeup. i've added it to the hashcat wiki pages. hope it's ok for you.

at 2011-10-31 08:44:32, chort wrote in to say...

That's cool with me. Thanks again atom!

Add a comment:




max length 1000 chars