Tuesday, November 29, 2011

Setting up a Password Cracking Server

By Brad Antoniewicz.

We do a ton of password cracking for our clients. Sometimes its because we've
compromised one system and are looking for password reuse, sometimes its because we're pulling password complexity statistics to prove a point, and sometimes we're just trying to break a WPA-PSK. Nonetheless it helps to have a beefy system on your side. We've recently revamped our cracking server, here is some information about its specifications and configuration.

Hardware Specs

EVGA Z68 FTW 160-SB-E689-KR Extended ATX Intel Motherboard

Intel Core i7-2600K 3.4GHz LGA 1155 95W Quad-Core Desktop Processor

CORSAIR CAFA50 120mm CPU Cooler
Power Supply
CORSAIR Professional Series Gold AX1200 (CMPSU-1200AX) 1200W ATX12V v2.31 / EPS12V v2.92 SLI Certified 80 PLUS GOLD Certified Modular Active PFC Power Supply

COOLER MASTER HAF 932 Advanced RC-932-KKN5-GP Black Steel ATX Full Tower Compucase Case with USB 3.0

2 - HITACHI Deskstar 3TB 3.5" SATA 6.0Gb/s Internal Hard Drive -Bare Drive

CORSAIR XMS3 16GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 Desktop Memory

4 - SAPPHIRE Radeon HD 6950 Dirt3 Edition 100312-3SR Video Card with Eyefinity

Total: $2,629.78 USD


Hardware Mods

Although there was a slot available on the motherboard, the 4th video card really didn't fit in with the others (Video card bullying), so we made the decision to mount it in a sort of "unique" way. I'm sure the air-flow/cooling Gods aren't happy with this decision, but it was the only option. Using a flexible PCIe extender, we mounted the card (cringe).

PCI Express Flexible Extender

You can see in the picture to the right, we dremel'ed out some holes in the back of the case to accommodate the connectors on the video card, and to provide some stability.

Hard drive/RAID Configuration

The 3TB drives required us to leverage Grub2 and GPT. Also it was important that we back up this system as best as we could in the case of hardware failure so we wanted to keep it simple with RAID1.

Intel FakeRAID via the BIOS - FAIL

Initially we wanted to use RAID 1 via the Intel RAID functionality built into the motherboard but it didn't seem like it was fully supported by dmraid or Ubuntu Server 10.04.3. For whatever reason it would only show up as a 801GB drive in parted and the installation wouldn't allow any partition to be created outside of that 801GB. So we abandoned it.

More fail info here: http://ubuntuforums.org/showthread.php?p=11477517

Software RAID (mdraid/mdadm) for the entire disk - FAIL

We wanted the drive to be fully redundant so we attempted to create a pretty large root partition and just a swap partition as linux_raid. The Ubuntu installation went fine with this attempt but the grub install failed. After a bunch of messing around we couldn't get grub properly installed. So we took a more traditional method of partitioning.

More fail info here: http://ubuntuforums.org/showthread.php?p=11477507

Software RAID (mdraid/mdadm) for root partition - WIN!

Here's how we ended up partitioning both (/dev/sda, /dev/sdb) drives. We did this before installation by booting to a LiveCD.

 root@ubuntu:~# parted /dev/sda mklabel gpt  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart grub 34 4130  
 root@ubuntu:~# parted -a cyl /dev/sdb set 1 bios_grub on  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart boot 4131 1052706  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart primary 1052707 5852144526  
 root@ubuntu:~# parted -a cyl /dev/sdb set 3 raid on  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart swap 5852144527s 5860533134s  
 root@ubuntu:~# parted /dev/sda unit compact print free  
 Model: ATA Hitachi HDS72303 (scsi)  
 Disk /dev/sda: 3001GB  
 Sector size (logical/physical): 512B/512B  
 Partition Table: gpt  
 Number Start  End   Size  File system   Name   Flags  
  1   17.4kB 2115kB 2098kB         grub   bios_grub  
  2   2115kB 539MB  537MB  ext2      boot  
  3   539MB  2996GB 2996GB xfs       primary raid  
  4   2996GB 3001GB 4295MB linux-swap(v1) swap  
Then within the installation, we configured RAID1 via mdraid for partition 3. Here's what the mdadm.conf looked like after the installation completed.
root@ubuntu:~# cat /etc/mdadm/mdadm.conf | grep ARRAY  
 ARRAY /dev/md0 level=raid1 metadata=1.0 num-devices=2 UUID=43dd56be:498abe6a:4421ece1:0659a71e name=0  
We selected package "Ubuntu Desktop" for GDM during the installation. This is important because later on the GPU cracking won't work without and X.org server running. Initially we had a startup script to just "xinit &" but that left a root shell for anyone with physical access to the system. Bad idea..

Once the installation finished, grub2 still didn't immediately boot, so we booted off of the LiveCD again and installed it manually:
root@ubuntu:~# mount -t xfs /dev/sda3 /mnt  
 root@ubuntu:~# cp /mnt/etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf  
 root@ubuntu:~# mdadm --assemble /dev/md0  
 mdadm: /dev/md0 has been started with 1 drive (out of 2).  
 root@ubuntu:~# umount /mnt  
 root@ubuntu:~# mount /dev/md0 /mnt  
 root@ubuntu:~# mount /dev/sda2 /mnt/boot  
 root@ubuntu:~# for i in /dev /dev/pts /proc /sys; do sudo mount -B $i /mnt$i; done  
 root@ubuntu:~# chroot /mnt  
 root@ubuntu:/# grub-mkdevicemap  
 root@ubuntu:/# grub-mkconfig -o /boot/grub/grub.cfg  
 Generating grub.cfg ...  
 Found linux image: /boot/vmlinuz-2.6.32-35-generic  
 Found initrd image: /boot/initrd.img-2.6.32-35-generic  
 Found memtest86+ image: /boot/memtest86+.bin  
 root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sda  
 Installation finished. No error reported.  
 root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sdb  
 Installation finished. No error reported.  
Then a reboot, and we're at a Ubuntu Gnome login prompt!

Installing OpenCL and AMD Drivers

Next we'll need to set up all those graphics cards. Since its AMD cards, we'll install the AMD OpenCL SDK and the AMD driver.
user@ubuntu:~$ mkdir AMD-APP-SDK-v2.5-lnx64
user@ubuntu:~$ cd AMD-APP-SDK-v2.5-lnx64/
user@ubuntu:~$ wget http://developer.amd.com/Downloads/AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ tar -zxvf AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ sudo ./Install-AMD-APP.sh
As a check, we can use the HelloCL test to make sure everything is working properly:

user@ubuntu:~$ cd /opt/AMDAPP/samples/opencl/bin/x86_64
user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ ./HelloCL
Getting Platform Information
No protocol specified
Creating a context AMD platform
Getting device info
Loading and compiling CL source
Running CL program
Looks good, now lets install the drivers.
user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ cd ~/
user@ubuntu:~$ sudo apt-get install libssl-dev build-essential python-dev libpcap-dev
user@ubuntu:~$ export AMDAPPSDKROOT="/opt/AMDAPP"
user@ubuntu:~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/opt/AMDAPP/lib/x86_64":"/opt/AMDAPP/lib/x86"
user@ubuntu:~$ chmod +x ati-driver-installer-11-11-x86.x86_64.run
user@ubuntu:~$ sudo ./ati-driver-installer-11-11-x86.x86_64.run

Installing Pyrit with OpenCL support

Pyrit is pretty awesome, but it doesn't take into account the latest AMD SDK. If you've been following the installation thus far, you should be ok. However you'll see that we'll have to make a couple changes in order to get everything working ok.
user@ubuntu:~$ wget http://pyrit.googlecode.com/files/cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ cd cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ vi setup.py
edit setup.py, create a new line after line 55 and add:
'/opt/AMDAPP/include', \
Once pyrit has all of the right includes, you can start the compilation/installation.
user@ubuntu:~/cpyrit-opencl-0.4.0$ python setup.py build
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install
user@ubuntu:~/cpyrit-opencl-0.4.0$ cd ..
user@ubuntu:~$ wget http://pyrit.googlecode.com/files/pyrit-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf pyrit-0.4.0.tar.gz
user@ubuntu:~$ cd pyrit-0.4.0
user@ubuntu:~/pyrit-0.4.0$ python setup.py build
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install
user@ubuntu:~/pyrit-0.4.0$ sudo pyrit list_cores
If you're using one video card, you should be all ok, however since we have 4, we have to use the aticonfig utility to enable them all:
user@ubuntu:~$ sudo service gdm stop
user@ubuntu:~$ sudo aticonfig --adapter=all --initial
user@ubuntu:~$ sudo service gdm start
We ran some quick pyrit benchmarks, things are looking pretty good:
user@ubuntu:~$ sudo pyrit benchmark
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (151933.5 PMKs/s)... /

Computed 151933.49 PMKs/s total.
#1: 'OpenCL-Device 'Cayman'': 41098.6 PMKs/s (RTT 2.7)
#2: 'OpenCL-Device 'Cayman'': 38496.3 PMKs/s (RTT 2.6)
#3: 'OpenCL-Device 'Cayman'': 42851.1 PMKs/s (RTT 2.6)
#4: 'OpenCL-Device 'Cayman'': 43023.6 PMKs/s (RTT 2.7)
#5: 'CPU-Core (SSE2)': 635.3 PMKs/s (RTT 3.5)
#6: 'CPU-Core (SSE2)': 588.7 PMKs/s (RTT 3.3)
#7: 'CPU-Core (SSE2)': 879.4 PMKs/s (RTT 2.8)
#8: 'CPU-Core (SSE2)': 961.1 PMKs/s (RTT 2.8)

Installing pyrit with CAL++ support

While 151933.5 PMKs/s is freaking awesome, we know that this isn't as fast as it *should* be. So instead of using the stable version of pyrit (0.4.0) with OpenCL, we decided to go with the SVN version of pyrit (0.4.1-dev) using CAL++.

The first thing we'll have to do is uninstall pyrit (trust me, its worth it):
user@ubuntu:~$ cd pyrit-0.4.0/
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)
user@ubuntu:~pyrit-0.4.0$ cd ../cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)
Now download CAL++ (http://sourceforge.net/projects/calpp/) and install it:
user@ubuntu:~$ sudo apt-get install libboost-all-dev cmake 
user@ubuntu:~$ tar -zxvf calpp-0.90.tar.gz
user@ubuntu:~$ cd calpp-0.90
CAL++ needs a couple of changes in order to properly find the AMD SDK libraries, so edit CMakeLists.txt and comment out (add a '#' to the beginning) these lines:
Add these:
Now we build and then install it:
user@ubuntu:~/calpp-0.90$ sudo make install
With CAL++ installed, you can download the development branch of pyrit and get the rest of the installation done.
user@ubuntu:~$ sudo apt-get install subversion
user@ubuntu:~$ svn checkout http://pyrit.googlecode.com/svn/trunk/ pyrit_svn
user@ubuntu:~$ cd pyrit_svn/cpyrit_calpp/
Did I mention this was a development version? Ya, well that means you'll have to make a couple changes:

edit setup.py line 28 to read:
VERSION = '0.4.1-dev'
edit setup.py line 35 to read:
edit setup.py line 41 to read:
CALPP_INC_DIRS.append(os.path.join(CALPP_INC_DIR, 'include/CAL'))
Alright, we're ready for the build and install:
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ python setup.py build
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ sudo python setup.py install
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ cd ../pyrit/
user@ubuntu:~/pyrit_svn/pyrit$ python setup.py build
user@ubuntu:~/pyrit_svn/pyritsudo python setup.py install
Scroll down to see the results below in the benchmarks section. We get an extra 20,000 PMK/s!


The oclHashcat install is extremely easy. Just decompress.

user@ubuntu:~$ sudo apt-get install p7zip
user@ubuntu:~$ p7zip -d oclHashcat-0.26.7z


user@ubuntu:~$ wget http://www.openwall.com/john/g/john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ tar -jxvf john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ cd john-1.7.8-jumbo-8/src/
user@ubuntu:~/john-1.7.8-jumbo-8/src$ make linux-x86-64

Next, we downloaded KoreLogic's Defcon 10 rulesets and added them to our john.conf
user@ubuntu:~/john-1.7.8-jumbo-8/src$ cd ../run
user@ubuntu:~/john-1.7.8-jumbo-8/run$ wget http://contest-2010.korelogic.com/rules.txt
user@ubuntu:~/john-1.7.8-jumbo-8/run$ cat rules.txt >> john.conf


Newer rainbow tables are .RTI files, meaning they're indexed rainbow tables, which are supposed to be faster then the normal ones. Since we have a handful of those, we'll need to install rcracki.

user@ubuntu:~$ wget http://www.freerainbowtables.com/download/rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ p7zip -d rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ cd rcracki_mt_0.6.6_src/

Edit the Makefile, and change the OPTIMIZATION variable to read:

OPTIMIZATION = -O3 -mtune=native

Now we can compile

user@ubuntu:~/rcracki_mt_0.6.6_src$ make
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo make install
user@ubuntu:~/rcracki_mt_0.6.6_src$ cd ..
user@ubuntu:~$ sudo mv rcracki_mt_0.6.6_src /cracking/bin/
user@ubuntu:~$ cd /cracking/bin/rcracki_mt_0.6.6_src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mkdir src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mv BaseRTReader.* Cha* COPYING CrackEngine.* fast_md* global.h Hash* INSTALLING.txt lm2ntlm.* Makefile MemoryPool.* Public.* RainbowCrack.* rcrackiThread.* README.txt RT* sha1.* TODO src/
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo chmod o+w rcracki.*


To support the older .RT rainbow tables, we can use standard rcrack. Version 1.5 comes pre-compiled, so the installation is really just a download and decompress.
user@ubuntu:~$ wget http://project-rainbowcrack.com/rainbowcrack-1.5-linux64.zip
user@ubuntu:~$ unzip rainbowcrack-1.5-linux64.zip


Rainbow Tables

What would a password cracking server, without a ton of Rainbow Tables. Luckily we have an entire of army of geeks at our disposal, so we didn't have to spend any time downloading. We copied over the following Rainbow Tables:
wpa-Offensive WPA Tables


Wordlists are one of those things that you can never seem to have enough of. We pulled together some popular ones, and I'm sure we'll be getting a lot of recommendations for others once the server reaches production.

user@ubuntu:/cracking/wordlists$ wget http://download.openwall.net/pub/wordlists/all.gz
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/500-worst-passwords.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/twitter-banned.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/english.txt.bz2
user@ubuntu:/cracking/wordlists$ bunzip2 *.bz2
user@ubuntu:/cracking/wordlists$ gunzip all.gz

Supporting a Multi-user environment

GPU Cracking

Since this system will be accessible by a bunch of people, I wanted to make sure no one stepped on eachother's toes. I wrote this simple wrapper so that if anyone tried to used oclHashcat or pyrit, while someone else is, they'd be told not to.

user@ubuntu:~$ cat /cracking/bin/gpu-crack

# gpu-crack - Simple GPU cracker wrapper for multi-user
#             environments
# by Brad Antoniewicz




echo -e "\n$0 v$ver"
echo "by brad a."
echo -e "-------------------------------------------------\n"

help() {
        echo "Usage:"
        echo -e "\t$0 [pyrit|hashcat] [options]\n"
        echo "Define what program you want to crack with (pyrit or hashcat)"
        echo "then provide the standard command line options that the cracker"
        echo "supports."

checkProc() {
        ps ax -o pid,user,etime,command | grep $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
        NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
        if [[ $NUMPROC != 0 ]]; then
                echo -e "[!] Found $NUMPROC instance of $1 running\n"

                for ((i=1; i<=$NUMPROC; i++))
#                        awk NR==$i "$TMPFILE"
                        PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
                        echo -e "\tPID: $PID"
                        echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2`"
                        echo -e "\tRunning for: `awk NR==$i "$TMPFILE" | awk '{print $3}'`"
                        echo -e "\n"
                echo -e "[-] No instances of $1 found"

launchProc() {
        for x in "$@"
                if [ $count != 0 ]; then
                        cmdlineArgs=$cmdlineArgs" "$x
                let count++

        echo "[+] Launching $1 with the following options"
        echo -e "\t $cmdlineArgs"

        if [[ $1 == "pyrit" ]]; then
                echo "[+] Running Pyrit"
                $SUDO_EXEC $PYRIT_EXEC $cmdlineArgs
        elif [[ $1 == "hashcat" ]]; then
                echo "[+] Running oclHashcat"
                $SUDO_EXEC $HASHCAT_EXEC $cmdlineArgs

if  [[ $# -ge 2 ]]  && [[  $1 == "pyrit" || $1 == "hashcat" ]]; then
        echo "[+] Checking for conflicting processes"
        checkProc "pyrit"
        checkProc "oclHash"

        if [ $isRunning == 0 ]; then
                launchProc $@
                echo "[!!] Found conflicting process, contact owner and make it stop!"

General Access

Since rcrack, rcracki, and john sort of need to be run from their program directories, I created a couple of aliases to get the user into the right spot with ease. Also, we put most of these installs into a central location (/cracking/bin) so that we can maintain them easy (no real packages for these tools).
user@ubuntu:~$ cat /etc/profile.d/pwserver.sh
# Set up all of the paths, directories, etc.. for password cracking
# by brad a.
alias john='cd /cracking/bin/john-bin; echo Changing directory - Please run john from here with ./john'
alias rcrack='cd /cracking/bin/rainbowcrack; echo -e "Changing directory - Please run rcrack from here with:\n./rcrack /path/to/tables/*.rt -f /path/to/pwdump"'
alias rcracki='cd /cracking/bin/rcracki_mt; echo -e "Changing directory - Please run rcracki from here with:\n./rcracki -f /path/to/pwdump /path/to/tables/"'

export PATH

Changing MOTD

What would be a Foundstone server without a fun MOTD. I've omitted the actual welcome message (stored in /etc/motd.tail) as it might be a little bit offensive :), but the stats script that gets launched on login is below.
user@ubuntu:~$ cd /etc/update-motd.d
user@ubuntu:/etc/update-motd.d$ sudo chmod -x *
user@ubuntu:/etc/update-motd.d$ sudo touch 10-pwserver
user@ubuntu:/etc/update-motd.d$ sudo chmod +x 10-pwserver
user@ubuntu:/etc/update-motd.d$ cat 10-pwserver

# by Brad Antoniewicz
[ -f /etc/motd.tail ] && cat /etc/motd.tail || true



processes="oclHashcat64.bin hashcat pyrit oclHash john rcrack rcracki rcracki_mt"

checkProc() {
        ps ax -o pid,user,etime,command | grep -w $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
        NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
        if [[ $NUMPROC != 0 ]]; then
                for ((i=1; i<=$NUMPROC; i++))
#                        awk NR==$i "$TMPFILE"
                        PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
                        echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2` - Process: $1 -  PID: $PID - Running For:  `awk NR==$i "$TMPFILE" | awk '{print $3}'`"

        echo Processes currently running
        for i in $processes
                checkProc $i

        if [ $isRunning == 0 ]; then
                echo -e "\tNone!"



Our pyrit benchmarks are decently respectable:
user@ubuntu:~$ gpu-crack pyrit benchmark

/usr/local/bin/gpu-crack v0.1by brad a.

[+] Checking for conflicting processes
[-] No instances of pyrit found
[-] No instances of oclHash found
[+] Launching pyrit with the following options
[+] Running Pyrit
Pyrit 0.4.1-dev (svn r308) (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (172644.8 PMKs/s)... /

Computed 172644.84 PMKs/s total.
#1: 'CAL++ Device #1 'AMD CAYMAN'': 51245.4 PMKs/s (RTT 1.0)
#2: 'CAL++ Device #2 'AMD CAYMAN'': 47993.0 PMKs/s (RTT 1.0)
#3: 'CAL++ Device #3 'AMD CAYMAN'': 51163.7 PMKs/s (RTT 1.0)
#4: 'CAL++ Device #4 'AMD CAYMAN'': 52829.2 PMKs/s (RTT 0.9)
#5: 'CPU-Core (SSE2/AES)': 713.9 PMKs/s (RTT 2.9)
#6: 'CPU-Core (SSE2/AES)': 736.3 PMKs/s (RTT 3.0)
#7: 'CPU-Core (SSE2/AES)': 821.7 PMKs/s (RTT 2.9)
#8: 'CPU-Core (SSE2/AES)': 763.8 PMKs/s (RTT 3.0)


oclHashcat looks good as well. One thing that's caught my attention is that HW Monitors 2-4 show 0% GPU. I'm not sure how to take that, but it's something I'll definitely need to investigate.
user@ubuntu:~$ sudo ./oclExample.sh
oclHashcat v0.26 by atom starting...

Digests: 6494 entries, 6494 unique
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes
Platform: AMD compatible platform found
Watchdog: Temperature limit set to 90c
Device #1: Cayman, 1024MB, 0Mhz, 22MCU
Device #2: Cayman, 1024MB, 0Mhz, 22MCU
Device #3: Cayman, 1024MB, 0Mhz, 22MCU
Device #4: Cayman, 1024MB, 0Mhz, 22MCU
NOTE: gpu-accel auto-adjusted to: 16
Device #1: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #2: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #3: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #4: Kernel ./kernels/4098/m0000.Cayman.64.kernel

--- CUT ----

[s]tatus [p]ause [r]esume [q]uit =>
Status.......: Exhausted
Hash.Type....: MD5
Mode.Left....: Mask '?l?l?l?l' (456976)
Mode.Right...: Dict 'example.dict' (129988)
Speed.GPU*...: 1761.3M/s
Recovered....: 1359/6494 Digests, 0/1 Salts
Progress.....: 59401396288/59401396288 (100.00%)
Running......: 18 secs
Estimated....: 0 secs
HW.Monitor.#1: 74% GPU, 71c Temp
HW.Monitor.#2:  0% GPU, 53c Temp
HW.Monitor.#3:  0% GPU, 43c Temp
HW.Monitor.#4:  0% GPU, 42c Temp

Started: Tue Nov 22 20:25:38 2011
Stopped: Tue Nov 22 20:25:56 2011


john only uses the CPUs, so no spectacular GPU stats here, but still some solid results.

john-1.7.8-jumbo-7/run$ ./john --test=30
Benchmarking: Traditional DES [128/128 BS SSE2-16]... DONE
Many salts: 4670K c/s real, 4685K c/s virtual
Only one salt: 3908K c/s real, 3920K c/s virtual

Benchmarking: BSDI DES (x725) [128/128 BS SSE2-16]... DONE
Many salts: 168806 c/s real, 169314 c/s virtual
Only one salt: 163686 c/s real, 164178 c/s virtual

Benchmarking: FreeBSD MD5 [8x]... DONE
Raw: 9768 c/s real, 9798 c/s virtual

Benchmarking: OpenBSD Blowfish (x32) [32/64 X2]... DONE
Raw: 870 c/s real, 873 c/s virtual

Benchmarking: Kerberos AFS DES [48/64 4K]... DONE
Short: 478762 c/s real, 480203 c/s virtual
Long: 1582K c/s real, 1587K c/s virtual

Benchmarking: LM DES [128/128 BS SSE2-16]... DONE
Raw: 25261K c/s real, 25337K c/s virtual

Benchmarking: Eggdrop [blowfish]... DONE
Raw: 29104 c/s real, 29191 c/s virtual

Benchmarking: DIGEST-MD5 [DIGEST-MD5 authentication]... DONE
Many salts: 1870K c/s real, 1876K c/s virtual
Only one salt: 1871K c/s real, 1876K c/s virtual

Benchmarking: More Secure Internet Password [RSA MD defined by BSAFE 1.x - Lotus v6]... DONE
Many salts: 101443 c/s real, 101748 c/s virtual
Only one salt: 60196 c/s real, 60378 c/s virtual

Benchmarking: EPiServer SID Hashes [SHA-1]... DONE
Many salts: 5542K c/s real, 5560K c/s virtual
Only one salt: 5257K c/s real, 5273K c/s virtual

Benchmarking: HTTP Digest access authentication [HDAA-MD5]... DONE
Many salts: 1988K c/s real, 1995K c/s virtual
Only one salt: 1940K c/s real, 1945K c/s virtual

Benchmarking: IPB2 MD5 [Invision Power Board 2.x salted MD5]... DONE
Many salts: 3988K c/s real, 4000K c/s virtual
Only one salt: 2485K c/s real, 2492K c/s virtual

Benchmarking: Kerberos v4 TGT [krb4 DES]... DONE
Raw: 3601K c/s real, 3612K c/s virtual

Benchmarking: Kerberos v5 TGT [krb5 3DES (des3-cbc-sha1)]... DONE
Raw: 63938 c/s real, 64130 c/s virtual

Benchmarking: MSCHAPv2 C/R MD4 DES [mschapv2]... DONE
Many salts: 3654K c/s real, 3666K c/s virtual
Only one salt: 2632K c/s real, 2640K c/s virtual

Benchmarking: MYSQL_fast [mysql-fast]... DONE
Raw: 42879K c/s real, 43009K c/s virtual

Benchmarking: MYSQL [mysql]... DONE
Raw: 5170K c/s real, 5185K c/s virtual

Benchmarking: LM C/R DES [netlm]... DONE
Many salts: 3621K c/s real, 3632K c/s virtual
Only one salt: 1216K c/s real, 1220K c/s virtual

Benchmarking: LMv2 C/R MD4 HMAC-MD5 [netlmv2]... DONE
Many salts: 1124K c/s real, 1127K c/s virtual
Only one salt: 908930 c/s real, 911665 c/s virtual

Benchmarking: NTLMv1 C/R MD4 DES [ESS MD5] [netntlm]... DONE
Many salts: 3651K c/s real, 3664K c/s virtual
Only one salt: 2652K c/s real, 2660K c/s virtual

Benchmarking: NTLMv2 C/R MD4 HMAC-MD5 [netntlmv2]... DONE
Many salts: 985779 c/s real, 989076 c/s virtual
Only one salt: 821367 c/s real, 823563 c/s virtual

Benchmarking: HalfLM C/R DES [nethalflm]... DONE
Many salts: 3622K c/s real, 3633K c/s virtual
Only one salt: 1825K c/s real, 1831K c/s virtual

Benchmarking: Netscape LDAP SSHA [salted SHA-1]... DONE
Many salts: 5197K c/s real, 5215K c/s virtual
Only one salt: 4738K c/s real, 4752K c/s virtual

Benchmarking: Netscape LDAP SHA [SHA-1]... DONE
Raw: 5029K c/s real, 5046K c/s virtual

Benchmarking: Netscreen MD5 [NS MD5]... DONE
Raw: 6159K c/s real, 6180K c/s virtual

Benchmarking: NT MD4 [128/128 X2 SSE2-16]... DONE
Raw: 47690K c/s real, 47849K c/s virtual

Benchmarking: OpenLDAP SSHA [salted SHA-1]... DONE
Many salts: 5281K c/s real, 5299K c/s virtual
Only one salt: 4883K c/s real, 4896K c/s virtual

Benchmarking: PHPS -- md5(md5($pass).$salt) [SSE2 16x4x2 (intr)]... DONE
Many salts: 9277K c/s real, 9305K c/s virtual
Only one salt: 3937K c/s real, 3949K c/s virtual

Benchmarking: Post.Office MD5 [STD]... DONE
Many salts: 4450K c/s real, 4463K c/s virtual
Only one salt: 4135K c/s real, 4146K c/s virtual

Benchmarking: Mac OS X 10.4 - 10.6 salted SHA-1 [32/64]... DONE
Many salts: 5886K c/s real, 5903K c/s virtual
Only one salt: 5575K c/s real, 5591K c/s virtual

Benchmarking: HMAC MD5 [hmac-md5]... DONE
Raw: 2043K c/s real, 2049K c/s virtual

Benchmarking: Lotus5 [Lotus v5 Proprietary]... DONE
Raw: 202395 c/s real, 203004 c/s virtual

Benchmarking: Generic salted MD4 [32/64]... DONE
Many salts: 9250K c/s real, 9278K c/s virtual
Only one salt: 8647K c/s real, 8673K c/s virtual

Benchmarking: MediaWiki -- md5($s.'-'.md5($p)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8465K c/s real, 8490K c/s virtual
Only one salt: 3815K c/s real, 3827K c/s virtual

Benchmarking: M$ Cache Hash [Generic 1x]... DONE
Many salts: 21450K c/s real, 21522K c/s virtual
Only one salt: 8711K c/s real, 8734K c/s virtual

Benchmarking: M$ Cache Hash 2 (DCC2) [SSE2-para 8x]... DONE
Raw: 402 c/s real, 403 c/s virtual

Benchmarking: MS Kerberos 5 AS-REQ Pre-Auth [mskrb5]... DONE
Many salts: 1153K c/s real, 1157K c/s virtual
Only one salt: 634491 c/s real, 636400 c/s virtual

Benchmarking: MS-SQL [mssql]... DONE
Many salts: 5505K c/s real, 5523K c/s virtual
Only one salt: 4844K c/s real, 4858K c/s virtual

Benchmarking: MS-SQL05 [ms-sql05]... DONE
Many salts: 5502K c/s real, 5521K c/s virtual
Only one salt: 5141K c/s real, 5156K c/s virtual

Benchmarking: MySQL 4.1 double-SHA-1 [mysql-sha1]... DONE
Raw: 2713K c/s real, 2722K c/s virtual

Benchmarking: Oracle 11g [oracle11]... DONE
Many salts: 5270K c/s real, 5287K c/s virtual
Only one salt: 4746K c/s real, 4759K c/s virtual

Benchmarking: Oracle [oracle]... DONE
Raw: 929803 c/s real, 932601 c/s virtual

Benchmarking: PHPass MD5 [SSE2 2x4x2 (intr)]... DONE
Raw: 4891 c/s real, 4905 c/s virtual

Benchmarking: PIX MD5 [pix-md5]... DONE
Raw: 6940K c/s real, 6963K c/s virtual

Benchmarking: pkzip [N/A]... DONE
Many salts: 10279K c/s real, 10314K c/s virtual
Only one salt: 7708K c/s real, 7731K c/s virtual

Benchmarking: Raw MD4 [32/64]... DONE
Raw: 9177K c/s real, 9205K c/s virtual

Benchmarking: Raw MD5 [SSE2 16x4x2 (intr)]... DONE
Raw: 8754K c/s real, 8780K c/s virtual

Benchmarking: Raw SHA-1(8x) [SHA-1]... DONE
Raw: 7107K c/s real, 7128K c/s virtual

Benchmarking: md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7508K c/s real, 7531K c/s virtual

Benchmarking: Salted SHA(8x) [SHA-1]... DONE
Many salts: 7616K c/s real, 7638K c/s virtual
Only one salt: 6608K c/s real, 6628K c/s virtual

Benchmarking: SAP BCODE [sapb]... DONE
Many salts: 2387K c/s real, 2394K c/s virtual
Only one salt: 2047K c/s real, 2054K c/s virtual

Benchmarking: SAP CODVN G (PASSCODE) [sapg]... DONE
Many salts: 1753K c/s real, 1758K c/s virtual
Only one salt: 1647K c/s real, 1651K c/s virtual

Benchmarking: Generic salted SHA-1 [32/64]... DONE
Many salts: 5175K c/s real, 5191K c/s virtual
Only one salt: 4947K c/s real, 4962K c/s virtual

Benchmarking: Tripcode DES [48/64 4K]... DONE
Raw: 383812 c/s real, 384967 c/s virtual

Benchmarking:  md5_gen(0): md5($p)  (raw-md5)  [SSE2 16x4x2 (intr)]... DONE
Raw: 8783K c/s real, 8809K c/s virtual

Benchmarking:  md5_gen(1): md5($p.$s)  (joomla)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6881K c/s real, 6901K c/s virtual

Benchmarking:  md5_gen(2): md5(md5($p))  (e107)  [SSE2 16x4x2 (intr)]... DONE
Raw: 4504K c/s real, 4517K c/s virtual

Benchmarking:  md5_gen(3): md5(md5(md5($p)))  [SSE2 16x4x2 (intr)]... DONE
Raw: 3028K c/s real, 3037K c/s virtual

Benchmarking:  md5_gen(4): md5($s.$p)  (OSC)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8218K c/s real, 8246K c/s virtual
Only one salt: 6975K c/s real, 6996K c/s virtual

Benchmarking:  md5_gen(5): md5($s.$p.$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 7621K c/s real, 7647K c/s virtual
Only one salt: 6508K c/s real, 6527K c/s virtual

Benchmarking:  md5_gen(6): md5(md5($p).$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 9179K c/s real, 9210K c/s virtual
Only one salt: 3980K c/s real, 3992K c/s virtual

Benchmarking:  md5_gen(8): md5(md5($s).$p)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8335K c/s real, 8363K c/s virtual
Only one salt: 7023K c/s real, 7044K c/s virtual

Benchmarking:  md5_gen(9): md5($s.md5($p))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8166K c/s real, 8193K c/s virtual
Only one salt: 3772K c/s real, 3783K c/s virtual

Benchmarking:  md5_gen(10): md5($s.md5($s.$p))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4129K c/s real, 4143K c/s virtual
Only one salt: 3774K c/s real, 3786K c/s virtual

Benchmarking:  md5_gen(11): md5($s.md5($p.$s))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4150K c/s real, 4164K c/s virtual
Only one salt: 3798K c/s real, 3809K c/s virtual

Benchmarking:  md5_gen(12): md5(md5($s).md5($p))  (IPB)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4408K c/s real, 4423K c/s virtual
Only one salt: 2623K c/s real, 2631K c/s virtual

Benchmarking:  md5_gen(13): md5(md5($p).md5($s))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4404K c/s real, 4419K c/s virtual
Only one salt: 2626K c/s real, 2634K c/s virtual

Benchmarking:  md5_gen(14): md5($s.md5($p).$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 7626K c/s real, 7649K c/s virtual
Only one salt: 3795K c/s real, 3806K c/s virtual

Benchmarking:  md5_gen(15): md5($u.md5($p).$s)  [64x2 (MD5_Body)]... DONE
Many salts: 9853K c/s real, 9883K c/s virtual
Only one salt: 4757K c/s real, 4772K c/s virtual

Benchmarking:  md5_gen(16): md5(md5(md5($p).$s).$s2)  [64x2 (MD5_Body)]... DONE
Many salts: 5344K c/s real, 5362K c/s virtual
Only one salt: 3170K c/s real, 3179K c/s virtual

Benchmarking:  md5_gen(17): phpass ($P$ or $H$)  [SSE2 2x4x2 (intr)]... DONE
Raw: 4902 c/s real, 4915 c/s virtual

Benchmarking:  md5_gen(18): md5($s.Y.$p.0xF7.$s) (Post.Office MD5)  [64x2 (MD5_Body)]... DONE
Many salts: 4341K c/s real, 4354K c/s virtual
Only one salt: 3981K c/s real, 3993K c/s virtual

Benchmarking:  md5_gen(19): Cisco PIX (MD5) [SSE2 16x4x2 (intr)]... DONE
Raw: 7481K c/s real, 7503K c/s virtual

Benchmarking:  md5_gen(20): Cisco PIX (MD5 salted) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8171K c/s real, 8196K c/s virtual
Only one salt: 6907K c/s real, 6928K c/s virtual

Benchmarking:  md5_gen(21): HTTP Digest Access Auth [SSE2 16x4x2 (intr)]... DONE
Many salts: 2027K c/s real, 2033K c/s virtual
Only one salt: 1960K c/s real, 1966K c/s virtual

Benchmarking:  md5_gen(22): md5(sha1($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3391K c/s real, 3401K c/s virtual

Benchmarking:  md5_gen(23): sha1(md5($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3223K c/s real, 3233K c/s virtual

Benchmarking:  md5_gen(24): sha1($p.$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 6076K c/s real, 6094K c/s virtual
Only one salt: 5242K c/s real, 5258K c/s virtual

Benchmarking:  md5_gen(25): sha1($s.$p) [SSE2 16x4x2 (intr)]... DONE
Many salts: 5946K c/s real, 5964K c/s virtual
Only one salt: 5315K c/s real, 5331K c/s virtual

Benchmarking:  md5_gen(26): sha1($p) raw-sha1 [4x2]... DONE
Raw: 6373K c/s real, 6393K c/s virtual

Benchmarking:  md5_gen(27): FreeBSD MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9502 c/s real, 9531 c/s virtual

Benchmarking:  md5_gen(28): Apache MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9456 c/s real, 9485 c/s virtual

Benchmarking:  md5_gen(29): md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7411K c/s real, 7436K c/s virtual

Benchmarking: md5_gen(1001) md5(md5(md5(md5($p)))) [SSE2 16x4x2 (intr)]... DONE
Raw: 2270K c/s real, 2278K c/s virtual

Benchmarking: md5_gen(1002) md5(md5(md5(md5(md5($p))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1823K c/s real, 1829K c/s virtual

Benchmarking: md5_gen(1003) md5(md5($p).md5($p)) [64x2 (MD5_Body)]... DONE
Raw: 2896K c/s real, 2905K c/s virtual

Benchmarking: md5_gen(1004) md5(md5(md5(md5(md5(md5($p)))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1523K c/s real, 1528K c/s virtual

Benchmarking: md5_gen(1005) md5(md5(md5(md5(md5(md5(md5($p))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1308K c/s real, 1312K c/s virtual

Benchmarking: md5_gen(1006) md5(md5(md5(md5(md5(md5(md5(md5($p)))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1145K c/s real, 1149K c/s virtual

Benchmarking: md5_gen(1008) md5($p.$s) [joomla] [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6868K c/s real, 6889K c/s virtual

Benchmarking: Raw SHA-224 [32/64]... DONE
Raw: 2577K c/s real, 2584K c/s virtual

Benchmarking: Raw SHA-256 [32/64]... DONE
Raw: 2600K c/s real, 2608K c/s virtual

Benchmarking: Raw SHA-384 [64/64]... DONE
Raw: 2054K c/s real, 2061K c/s virtual

Benchmarking: Raw SHA-512 [64/64]... DONE
Raw: 2054K c/s real, 2060K c/s virtual

Benchmarking: Mac OS X 10.7+ salted SHA-512 [64/64]... DONE
Many salts: 2098K c/s real, 2104K c/s virtual
Only one salt: 2056K c/s real, 2062K c/s virtual

Benchmarking: hmailserver [32/64]... DONE
Many salts: 2607K c/s real, 2614K c/s virtual
Only one salt: 2488K c/s real, 2495K c/s virtual

Benchmarking: sybasease [32/64]... DONE
Many salts: 344041 c/s real, 345076 c/s virtual
Only one salt: 341478 c/s real, 342505 c/s virtual

Benchmarking: generic crypt(3) [?/64]... DONE
Many salts: 405132 c/s real, 406351 c/s virtual
Only one salt: 403443 c/s real, 404657 c/s virtual

Benchmarking: ssh [32/64]... DONE
Raw: 268979 c/s real, 269878 c/s virtual

Benchmarking: pdf [32/64]... DONE
Many salts: 22014 c/s real, 22087 c/s virtual
Only one salt: 45401 c/s real, 45537 c/s virtual

Benchmarking: rar [32/64]... DONE
Raw: 48.2 c/s real, 48.3 c/s virtual

Benchmarking: zip [32/64]... DONE
Raw: 766 c/s real, 769 c/s virtual

Benchmarking: dummy [N/A]... DONE
Raw: 126288K c/s real, 126710K c/s virtual

Wrap up

Like many installations, we ran into a couple of road bumps starting out, but now our server is looking good. Before we make it live, we'll lock down the user permissions, roles, and set up a few cron jobs to keep the system tidy.

What do you think? Got any tips for our rig? Have a few pointers from your own experiences? Drop us some info in the comments below.


  1. Nice article and a beast of a server!

    Have you ever tried password cracking via Amazon EC2? If so how did it compare to your own password cracking server?

  2. This is a confusing. You are testing WPA/WPA2 cracking with pyrit but oclHashcat-plus is the faster WPA/WPA2 cracker. You show pyrit is around 151k, oclHashcat-plus will be around 270k !!! Plus it will not use your CPU. So you can use it for JtR or just save some energy. Next, you are testing MD5 cracking with oclHashcat but oclHashcat-lite is the faster MD5 cracker. You show oclHashcat is around 1750M but oclHashcat-lite will do 18500M or more !!! OK, its a single hash cracker, but much better suited for a benchmark. About the Amazon EC2: Its simple. They suck for password cracking. If you want fast password cracking, go red.

  3. go QUAD x 6990 / or QUAD x 5970. Thats the way to go.



  4. I believe AMD app sdk 2.5 and the amd 11.11 drivers are not compatible with hashcat (which is why only one of the cards works). They recommend sdk 2.4 and 11.4 drivers.

  5. That's cool but can it play Crysis?

    On a slightly more serious note, how come you decided to buy all Corsair where available as opposed to a leading brand/product from each component category? And what clock are you running your 2600K for? If not OCing CPU, why did you opt to buty the K version? Hoping for a higher-binned chip?

  6. Avid GPGPU enthusiast here:

    Looks good but you really need to re-think your cooling solutions. I had heat-soak issues when I started to overclock. Additionally, you really need to think about having 2 PSU's instead of one big one, much more stable that way. I like how you went with Nvidia for the mutli-job based cracking capability. However, if you are only going to be doing one job at a time, ATI cards are superior because of how they handle low-level algorithms. Keep a restored image of your current OS with hardware settings backed up, an KEEP TWEAKING SETTINGS! You will be surprised what a simple driver upgrade (or downgrade) can change. You should be pulling Much higher numbers in some areas, but still, looks like you know what you are doing. Keep it up! cheers.

  7. I also was wondering about leasing time on EC2 rather than building a dedicated setup. I'm guessing your hardware setup won't really be practically useful past 24 months due to improving hardware, so the depreciation schedule's even worse than for normal IT equipment, making Amazon's offerings possibly more attractive.

    Have you run the numbers on your average utilization to see if the number of hours you'll be running it make owning worthwhile?

    @atom, are you referring to Amazon's high performance computing/cluster GPU offerings, or just the standard stuff?

    The HPC stuff seems pretty reasonable:

    $2.40/hr for dual eight core Xeons or $2.10/hr for dual quad core Nehalem Xeons with dual Nvidia M2050s.

  8. Building the cracker jack..

  9. @Anonymous: Nothing against Amazon's high performance computing/cluster GPU, but they work with NVidia. For some other tasks this might be a good choice but for hash cracking there is no way around ATI/AMD. Their GPU's have instructions for BFI_INT and BIT_ALIGN, which are heavily used in hashing algorithms. This way they are more then superior to NVidia. Take a look a the benchmarks on the oclHashcat-lite site.

  10. For a comparison with EC2 see the second half of http://blog.zorinaq.com/?e=42

  11. Nice article man! thats slick!

  12. Brad if your running into heat issues you may need to set up a script to set your fan speeds. I've found that by default they are at 20%, and linux does not auto adjust the speed. The script I use for "Erebus" can be downloaded here: http://ob-security.info/files/fan.sh
    You will need to adjust the fanspeed % depending on what temperature each card is at.

  13. My favorite wordlists are here:

    g0tmi1k did a great job combining, sorting, and cleaning out the lists. Search the page for "18-in-1"

  14. Why didn't you use the HAF-X case? I'm not 100% certain, but I'm guessing it'd fit all your cards a lot easier and provide some extra breathing room. All for about the same price as the case you bought.

  15. I nice setup you should look at utilizing the Moscrack app and build a mpi cluster as Moscrack will run over a cluster enviroment and has a pyrit plugin that will allow you to use multiple nodes with your sam setup. This is what i have been working on for a while know hopefully looking at around 1million pmks/sec when its finished over a 10 node cluster yes its expensive to build but FUN


  16. Thank u very much guys it was a long way but finaly I managed get it to work:
    pyrit ,call++ ,amd beta 12.11 drivers ,hd7970 here is an output:

    Running benchmark (112016.0 PMKs/s)... /

    Computed 112016.03 PMKs/s total.
    #1: 'CAL++ Device #1 'AMD GPU DEVICE'': 117430.1 PMKs/s (RTT 1.0)
    #2: 'CPU-Core (SSE2)': 411.5 PMKs/s (RTT 3.2)
    #3: 'CPU-Core (SSE2)': 409.3 PMKs/s (RTT 3.0)
    #4: 'CPU-Core (SSE2)': 402.2 PMKs/s (RTT 3.2)
    #5: 'CPU-Core (SSE2)': 403.7 PMKs/s (RTT 3.0)
    #6: 'CPU-Core (SSE2)': 405.4 PMKs/s (RTT 3.0)
    #7: 'CPU-Core (SSE2)': 408.7 PMKs/s (RTT 3.1)
    #8: 'CPU-Core (SSE2)': 440.6 PMKs/s (RTT 3.0)

    more than happy with that :)))

    atom thank u for ur advices oclhashcat-plus doing better but I didn't find way how to limit the number of consecutive identical characters in bruteforce even pipe it with crunch coz I'm usin win7 for that.
    all the best guys