Setting up a Password Cracking Server

By Brad Antoniewicz.

We do a ton of password cracking for our clients. Sometimes its because we've
compromised one system and are looking for password reuse, sometimes its because we're pulling password complexity statistics to prove a point, and sometimes we're just trying to break a WPA-PSK. Nonetheless it helps to have a beefy system on your side. We've recently revamped our cracking server, here is some information about its specifications and configuration.

Hardware Specs

Motherboard EVGA Z68 FTW 160-SB-E689-KR Extended ATX Intel Motherboard
CPU Intel Core i7-2600K 3.4GHz LGA 1155 95W Quad-Core Desktop Processor
CORSAIR CAFA50 120mm CPU Cooler
Power Supply CORSAIR Professional Series Gold AX1200 (CMPSU-1200AX) 1200W ATX12V v2.31 / EPS12V v2.92 SLI Certified 80 PLUS GOLD Certified Modular Active PFC Power Supply
Case COOLER MASTER HAF 932 Advanced RC-932-KKN5-GP Black Steel ATX Full Tower Compucase Case with USB 3.0
Storage 2 - HITACHI Deskstar 3TB 3.5" SATA 6.0Gb/s Internal Hard Drive -Bare Drive
Memory CORSAIR XMS3 16GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 Desktop Memory
GPU 4 - SAPPHIRE Radeon HD 6950 Dirt3 Edition 100312-3SR Video Card with Eyefinity

Total: $2,629.78 USD

Installation

Hardware Mods

Although there was a slot available on the motherboard, the 4th video card really didn't fit in with the others (Video card bullying), so we made the decision to mount it in a sort of "unique" way. I'm sure the air-flow/cooling Gods aren't happy with this decision, but it was the only option. Using a flexible PCIe extender, we mounted the card (cringe).

PCI Express Flexible Extender
You can see in the picture to the right, we dremel'ed out some holes in the back of the case to accommodate the connectors on the video card, and to provide some stability.

Hard drive/RAID Configuration

The 3TB drives required us to leverage Grub2 and GPT. Also it was important that we back up this system as best as we could in the case of hardware failure so we wanted to keep it simple with RAID1.

Intel FakeRAID via the BIOS - FAIL

Initially we wanted to use RAID 1 via the Intel RAID functionality built into the motherboard but it didn't seem like it was fully supported by dmraid or Ubuntu Server 10.04.3. For whatever reason it would only show up as a 801GB drive in parted and the installation wouldn't allow any partition to be created outside of that 801GB. So we abandoned it.

More fail info here: http://ubuntuforums.org/showthread.php?p=11477517

Software RAID (mdraid/mdadm) for the entire disk - FAIL

We wanted the drive to be fully redundant so we attempted to create a pretty large root partition and just a swap partition as linux_raid. The Ubuntu installation went fine with this attempt but the grub install failed. After a bunch of messing around we couldn't get grub properly installed. So we took a more traditional method of partitioning.

More fail info here: http://ubuntuforums.org/showthread.php?p=11477507

Software RAID (mdraid/mdadm) for root partition - WIN!

Here's how we ended up partitioning both (/dev/sda, /dev/sdb) drives. We did this before installation by booting to a LiveCD.

 root@ubuntu:~# parted /dev/sda mklabel gpt  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart grub 34 4130  
 root@ubuntu:~# parted -a cyl /dev/sdb set 1 bios_grub on  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart boot 4131 1052706  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart primary 1052707 5852144526  
 root@ubuntu:~# parted -a cyl /dev/sdb set 3 raid on  
 root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart swap 5852144527s 5860533134s  
 root@ubuntu:~# parted /dev/sda unit compact print free  
 Model: ATA Hitachi HDS72303 (scsi)  
 Disk /dev/sda: 3001GB  
 Sector size (logical/physical): 512B/512B  
 Partition Table: gpt  
 Number Start  End   Size  File system   Name   Flags  
  1   17.4kB 2115kB 2098kB         grub   bios_grub  
  2   2115kB 539MB  537MB  ext2      boot  
  3   539MB  2996GB 2996GB xfs       primary raid  
  4   2996GB 3001GB 4295MB linux-swap(v1) swap  

Then within the installation, we configured RAID1 via mdraid for partition 3. Here's what the mdadm.conf looked like after the installation completed.

root@ubuntu:~# cat /etc/mdadm/mdadm.conf | grep ARRAY  
 ARRAY /dev/md0 level=raid1 metadata=1.0 num-devices=2 UUID=43dd56be:498abe6a:4421ece1:0659a71e name=0  

We selected package "Ubuntu Desktop" for GDM during the installation. This is important because later on the GPU cracking won't work without and X.org server running. Initially we had a startup script to just "xinit &" but that left a root shell for anyone with physical access to the system. Bad idea..

Once the installation finished, grub2 still didn't immediately boot, so we booted off of the LiveCD again and installed it manually:

root@ubuntu:~# mount -t xfs /dev/sda3 /mnt  
 root@ubuntu:~# cp /mnt/etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf  
 root@ubuntu:~# mdadm --assemble /dev/md0  
 mdadm: /dev/md0 has been started with 1 drive (out of 2).  
 root@ubuntu:~# umount /mnt  
 root@ubuntu:~# mount /dev/md0 /mnt  
 root@ubuntu:~# mount /dev/sda2 /mnt/boot  
 root@ubuntu:~# for i in /dev /dev/pts /proc /sys; do sudo mount -B $i /mnt$i; done  
 root@ubuntu:~# chroot /mnt  
 root@ubuntu:/# grub-mkdevicemap  
 root@ubuntu:/# grub-mkconfig -o /boot/grub/grub.cfg  
 Generating grub.cfg ...  
 Found linux image: /boot/vmlinuz-2.6.32-35-generic  
 Found initrd image: /boot/initrd.img-2.6.32-35-generic  
 Found memtest86+ image: /boot/memtest86+.bin  
 root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sda  
 Installation finished. No error reported.  
 root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sdb  
 Installation finished. No error reported.  

Then a reboot, and we're at a Ubuntu Gnome login prompt!

Installing OpenCL and AMD Drivers

Next we'll need to set up all those graphics cards. Since its AMD cards, we'll install the AMD OpenCL SDK and the AMD driver.

user@ubuntu:~$ mkdir AMD-APP-SDK-v2.5-lnx64
user@ubuntu:~$ cd AMD-APP-SDK-v2.5-lnx64/
user@ubuntu:~$ wget http://developer.amd.com/Downloads/AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ tar -zxvf AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ sudo ./Install-AMD-APP.sh

As a check, we can use the HelloCL test to make sure everything is working properly:

user@ubuntu:~$ cd /opt/AMDAPP/samples/opencl/bin/x86_64
user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ ./HelloCL
HelloCL!
Getting Platform Information
No protocol specified
Creating a context AMD platform
Getting device info
Loading and compiling CL source
Running CL program
Done
Passed!

Looks good, now lets install the drivers.

user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ cd ~/
user@ubuntu:~$ sudo apt-get install libssl-dev build-essential python-dev libpcap-dev
user@ubuntu:~$ export AMDAPPSDKROOT="/opt/AMDAPP"
user@ubuntu:~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/opt/AMDAPP/lib/x86_64":"/opt/AMDAPP/lib/x86"
user@ubuntu:~$ export ATISTREAMSDKROOT=$AMDAPPSDKROOT
user@ubuntu:~$ chmod +x ati-driver-installer-11-11-x86.x86_64.run
user@ubuntu:~$ sudo ./ati-driver-installer-11-11-x86.x86_64.run

Installing Pyrit with OpenCL support

Pyrit is pretty awesome, but it doesn't take into account the latest AMD SDK. If you've been following the installation thus far, you should be ok. However you'll see that we'll have to make a couple changes in order to get everything working ok.

user@ubuntu:~$ wget http://pyrit.googlecode.com/files/cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ cd cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ vi setup.py

edit setup.py, create a new line after line 55 and add:

'/opt/AMDAPP/include', \

Once pyrit has all of the right includes, you can start the compilation/installation.

user@ubuntu:~/cpyrit-opencl-0.4.0$ python setup.py build
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install
user@ubuntu:~/cpyrit-opencl-0.4.0$ cd ..
user@ubuntu:~$ wget http://pyrit.googlecode.com/files/pyrit-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf pyrit-0.4.0.tar.gz
user@ubuntu:~$ cd pyrit-0.4.0
user@ubuntu:~/pyrit-0.4.0$ python setup.py build
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install
user@ubuntu:~/pyrit-0.4.0$ sudo pyrit list_cores

If you're using one video card, you should be all ok, however since we have 4, we have to use the aticonfig utility to enable them all:

user@ubuntu:~$ sudo service gdm stop
user@ubuntu:~$ sudo aticonfig --adapter=all --initial
user@ubuntu:~$ sudo service gdm start

We ran some quick pyrit benchmarks, things are looking pretty good:

user@ubuntu:~$ sudo pyrit benchmark
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (151933.5 PMKs/s)... /

Computed 151933.49 PMKs/s total.
#1: 'OpenCL-Device 'Cayman'': 41098.6 PMKs/s (RTT 2.7)
#2: 'OpenCL-Device 'Cayman'': 38496.3 PMKs/s (RTT 2.6)
#3: 'OpenCL-Device 'Cayman'': 42851.1 PMKs/s (RTT 2.6)
#4: 'OpenCL-Device 'Cayman'': 43023.6 PMKs/s (RTT 2.7)
#5: 'CPU-Core (SSE2)': 635.3 PMKs/s (RTT 3.5)
#6: 'CPU-Core (SSE2)': 588.7 PMKs/s (RTT 3.3)
#7: 'CPU-Core (SSE2)': 879.4 PMKs/s (RTT 2.8)
#8: 'CPU-Core (SSE2)': 961.1 PMKs/s (RTT 2.8)

Installing pyrit with CAL++ support

While 151933.5 PMKs/s is freaking awesome, we know that this isn't as fast as it *should* be. So instead of using the stable version of pyrit (0.4.0) with OpenCL, we decided to go with the SVN version of pyrit (0.4.1-dev) using CAL++.

The first thing we'll have to do is uninstall pyrit (trust me, its worth it):

user@ubuntu:~$ cd pyrit-0.4.0/
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)
user@ubuntu:~pyrit-0.4.0$ cd ../cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)

Now download CAL++ (http://sourceforge.net/projects/calpp/) and install it:

user@ubuntu:~$ sudo apt-get install libboost-all-dev cmake 
user@ubuntu:~$ tar -zxvf calpp-0.90.tar.gz
user@ubuntu:~$ cd calpp-0.90

CAL++ needs a couple of changes in order to properly find the AMD SDK libraries, so edit CMakeLists.txt and comment out (add a '#' to the beginning) these lines:

FIND_LIBRARY( LIB_ATICALCL aticalcl PATHS "$ENV{ATISTREAMSDKROOT}/lib" )
FIND_LIBRARY( LIB_ATICALRT aticalrt PATHS "$ENV{ATISTREAMSDKROOT}/lib" )
FIND_PATH( LIB_ATICAL_INCLUDE NAMES cal.h calcl.h PATHS "$ENV{ATISTREAMSDKROOT}/include" )

Add these:

FIND_LIBRARY( LIB_ATICALCL aticalcl PATHS "/opt/AMDAPP/lib" )
FIND_LIBRARY( LIB_ATICALRT aticalrt PATHS "/opt/AMDAPP/lib" )
FIND_PATH( LIB_ATICAL_INCLUDE NAMES cal.h calcl.h PATHS "/opt/AMDAPP/include/CAL" )

Now we build and then install it:

user@ubuntu:~/calpp-0.90$ sudo make install

With CAL++ installed, you can download the development branch of pyrit and get the rest of the installation done.

user@ubuntu:~$ sudo apt-get install subversion
user@ubuntu:~$ svn checkout http://pyrit.googlecode.com/svn/trunk/ pyrit_svn
user@ubuntu:~$ cd pyrit_svn/cpyrit_calpp/

Did I mention this was a development version? Ya, well that means you'll have to make a couple changes:

edit setup.py line 28 to read:

VERSION = '0.4.1-dev'

edit setup.py line 35 to read:

CALPP_INC_DIR = '/opt/AMDAPP'

edit setup.py line 41 to read:

CALPP_INC_DIRS.append(os.path.join(CALPP_INC_DIR, 'include/CAL'))

Alright, we're ready for the build and install:

user@ubuntu:~/pyrit_svn/cpyrit_calpp$ python setup.py build
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ sudo python setup.py install
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ cd ../pyrit/
user@ubuntu:~/pyrit_svn/pyrit$ python setup.py build
user@ubuntu:~/pyrit_svn/pyritsudo python setup.py install

Scroll down to see the results below in the benchmarks section. We get an extra 20,000 PMK/s!

oclHashcat

The oclHashcat install is extremely easy. Just decompress.

user@ubuntu:~$ sudo apt-get install p7zip
user@ubuntu:~$ p7zip -d oclHashcat-0.26.7z

john

user@ubuntu:~$ wget http://www.openwall.com/john/g/john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ tar -jxvf john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ cd john-1.7.8-jumbo-8/src/
user@ubuntu:~/john-1.7.8-jumbo-8/src$ make linux-x86-64

Next, we downloaded KoreLogic's Defcon 10 rulesets and added them to our john.conf

user@ubuntu:~/john-1.7.8-jumbo-8/src$ cd ../run
user@ubuntu:~/john-1.7.8-jumbo-8/run$ wget http://contest-2010.korelogic.com/rules.txt
user@ubuntu:~/john-1.7.8-jumbo-8/run$ cat rules.txt >> john.conf

rcracki

Newer rainbow tables are .RTI files, meaning they're indexed rainbow tables, which are supposed to be faster then the normal ones. Since we have a handful of those, we'll need to install rcracki.

user@ubuntu:~$ wget http://www.freerainbowtables.com/download/rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ p7zip -d rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ cd rcracki_mt_0.6.6_src/

Edit the Makefile, and change the OPTIMIZATION variable to read:

OPTIMIZATION = -O3 -mtune=native

Now we can compile

user@ubuntu:~/rcracki_mt_0.6.6_src$ make
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo make install
user@ubuntu:~/rcracki_mt_0.6.6_src$ cd ..
user@ubuntu:~$ sudo mv rcracki_mt_0.6.6_src /cracking/bin/
user@ubuntu:~$ cd /cracking/bin/rcracki_mt_0.6.6_src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mkdir src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mv BaseRTReader.* Cha* COPYING CrackEngine.* fast_md* global.h Hash* INSTALLING.txt lm2ntlm.* Makefile MemoryPool.* Public.* RainbowCrack.* rcrackiThread.* README.txt RT* sha1.* TODO src/
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo chmod o+w rcracki.*

rcrack

To support the older .RT rainbow tables, we can use standard rcrack. Version 1.5 comes pre-compiled, so the installation is really just a download and decompress.

user@ubuntu:~$ wget http://project-rainbowcrack.com/rainbowcrack-1.5-linux64.zip
user@ubuntu:~$ unzip rainbowcrack-1.5-linux64.zip

Downloads

Rainbow Tables

What would a password cracking server, without a ton of Rainbow Tables. Luckily we have an entire of army of geeks at our disposal, so we didn't have to spend any time downloading. We copied over the following Rainbow Tables:

fastlm_alpha-numeric#1-7_0
lm_all-space_1-7
lm_alpha_1-7
lm_alpha-numeric_1-7
lm_alpha-numeric-symbol32-space_1-7
ntlm-alpha-numeric-space_1-8
ntlm_alpha-space_1-9
ntlm_hybrid(loweralpha#6-6,numeric#1-3)
ntlm_hybrid(loweralpha#7-7,numeric#1-3)
ntlm_loweralpha-numeric-space_1-8
ntlm_loweralpha-numeric-symbol32-space_1-7_0
ntlm_loweralpha-space_1-9
ntlm_mixalpha-numeric_1-8
ntlm_mixalpha-numeric-all-space_1-6
ntlm_mixalpha-numeric-space_1-7_0
ntlm_numeric_1-12
Ophcrack_Tables-LM_Fast_Free
Ophcrack_Tables-NTLM_Free
sha1_loweralpha-numeric-space_1-8
wpa-h1kari_renderman
wpa-Offensive WPA Tables

Wordlists

Wordlists are one of those things that you can never seem to have enough of. We pulled together some popular ones, and I'm sure we'll be getting a lot of recommendations for others once the server reaches production.

user@ubuntu:/cracking/wordlists$ wget http://download.openwall.net/pub/wordlists/all.gz
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/500-worst-passwords.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/twitter-banned.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/english.txt.bz2
user@ubuntu:/cracking/wordlists$ bunzip2 *.bz2
user@ubuntu:/cracking/wordlists$ gunzip all.gz

Supporting a Multi-user environment

GPU Cracking

Since this system will be accessible by a bunch of people, I wanted to make sure no one stepped on eachother's toes. I wrote this simple wrapper so that if anyone tried to used oclHashcat or pyrit, while someone else is, they'd be told not to.

user@ubuntu:~$ cat /cracking/bin/gpu-crack
#!/bin/bash

#
# gpu-crack - Simple GPU cracker wrapper for multi-user
#             environments
#
# by Brad Antoniewicz
#

TMPFILE="/tmp/check.tmp"
ver="0.1"

SUDO_EXEC=/usr/bin/sudo
PYRIT_EXEC=/usr/local/bin/pyrit.real
HASHCAT_EXEC=/cracking/bin/oclhash/oclHashcat64.bin

isRunning=0;


echo -e "\n$0 v$ver"
echo "by brad a."
echo -e "-------------------------------------------------\n"


help() {
        echo "Usage:"
        echo -e "\t$0 [pyrit|hashcat] [options]\n"
        echo "Define what program you want to crack with (pyrit or hashcat)"
        echo "then provide the standard command line options that the cracker"
        echo "supports."
}

checkProc() {
        ps ax -o pid,user,etime,command | grep $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
        NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
        if [[ $NUMPROC != 0 ]]; then
                echo -e "[!] Found $NUMPROC instance of $1 running\n"

                for ((i=1; i<=$NUMPROC; i++))
                do
#                        awk NR==$i "$TMPFILE"
                        PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
                        echo -e "\tPID: $PID"
                        echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2`"
                        echo -e "\tRunning for: `awk NR==$i "$TMPFILE" | awk '{print $3}'`"
                        echo -e "\n"
                        isRunning=1;
                done
        else
                echo -e "[-] No instances of $1 found"
        fi
}

launchProc() {
        count=0;
        for x in "$@"
        do
                if [ $count != 0 ]; then
                        cmdlineArgs=$cmdlineArgs" "$x
                fi
                let count++
        done

        echo "[+] Launching $1 with the following options"
        echo -e "\t $cmdlineArgs"

        if [[ $1 == "pyrit" ]]; then
                echo "[+] Running Pyrit"
                $SUDO_EXEC $PYRIT_EXEC $cmdlineArgs
        elif [[ $1 == "hashcat" ]]; then
                echo "[+] Running oclHashcat"
                $SUDO_EXEC $HASHCAT_EXEC $cmdlineArgs
        fi
}

if  [[ $# -ge 2 ]]  && [[  $1 == "pyrit" || $1 == "hashcat" ]]; then
        echo "[+] Checking for conflicting processes"
        checkProc "pyrit"
        checkProc "oclHash"

        if [ $isRunning == 0 ]; then
                launchProc $@
        else
                echo "[!!] Found conflicting process, contact owner and make it stop!"
        fi
else
        help
fi

General Access

Since rcrack, rcracki, and john sort of need to be run from their program directories, I created a couple of aliases to get the user into the right spot with ease. Also, we put most of these installs into a central location (/cracking/bin) so that we can maintain them easy (no real packages for these tools).

user@ubuntu:~$ cat /etc/profile.d/pwserver.sh
#
# Set up all of the paths, directories, etc.. for password cracking
# by brad a.
#
alias john='cd /cracking/bin/john-bin; echo Changing directory - Please run john from here with ./john'
alias rcrack='cd /cracking/bin/rainbowcrack; echo -e "Changing directory - Please run rcrack from here with:\n./rcrack /path/to/tables/*.rt -f /path/to/pwdump"'
alias rcracki='cd /cracking/bin/rcracki_mt; echo -e "Changing directory - Please run rcracki from here with:\n./rcracki -f /path/to/pwdump /path/to/tables/"'

PATH=$PATH:/cracking/bin
export PATH

Changing MOTD

What would be a Foundstone server without a fun MOTD. I've omitted the actual welcome message (stored in /etc/motd.tail) as it might be a little bit offensive :), but the stats script that gets launched on login is below.

user@ubuntu:~$ cd /etc/update-motd.d
user@ubuntu:/etc/update-motd.d$ sudo chmod -x *
user@ubuntu:/etc/update-motd.d$ sudo touch 10-pwserver
user@ubuntu:/etc/update-motd.d$ sudo chmod +x 10-pwserver
user@ubuntu:/etc/update-motd.d$ cat 10-pwserver
#!/bin/bash

#
#
# by Brad Antoniewicz
#
[ -f /etc/motd.tail ] && cat /etc/motd.tail || true


TMPFILE="/tmp/check.tmp"
ver="0.1"

isRunning=0;

processes="oclHashcat64.bin hashcat pyrit oclHash john rcrack rcracki rcracki_mt"


checkProc() {
        ps ax -o pid,user,etime,command | grep -w $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
        NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
        if [[ $NUMPROC != 0 ]]; then
                for ((i=1; i<=$NUMPROC; i++))
                do
#                        awk NR==$i "$TMPFILE"
                        PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
                        echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2` - Process: $1 -  PID: $PID - Running For:  `awk NR==$i "$TMPFILE" | awk '{print $3}'`"
                        isRunning=1;
                done
        fi
}

        echo Processes currently running
        for i in $processes
        do
                checkProc $i
        done

        if [ $isRunning == 0 ]; then
                echo -e "\tNone!"
        fi

Benchmarks

pyrit

Our pyrit benchmarks are decently respectable:

user@ubuntu:~$ gpu-crack pyrit benchmark

/usr/local/bin/gpu-crack v0.1by brad a.
-------------------------------------------------

[+] Checking for conflicting processes
[-] No instances of pyrit found
[-] No instances of oclHash found
[+] Launching pyrit with the following options
          benchmark
[+] Running Pyrit
Pyrit 0.4.1-dev (svn r308) (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+

Running benchmark (172644.8 PMKs/s)... /

Computed 172644.84 PMKs/s total.
#1: 'CAL++ Device #1 'AMD CAYMAN'': 51245.4 PMKs/s (RTT 1.0)
#2: 'CAL++ Device #2 'AMD CAYMAN'': 47993.0 PMKs/s (RTT 1.0)
#3: 'CAL++ Device #3 'AMD CAYMAN'': 51163.7 PMKs/s (RTT 1.0)
#4: 'CAL++ Device #4 'AMD CAYMAN'': 52829.2 PMKs/s (RTT 0.9)
#5: 'CPU-Core (SSE2/AES)': 713.9 PMKs/s (RTT 2.9)
#6: 'CPU-Core (SSE2/AES)': 736.3 PMKs/s (RTT 3.0)
#7: 'CPU-Core (SSE2/AES)': 821.7 PMKs/s (RTT 2.9)
#8: 'CPU-Core (SSE2/AES)': 763.8 PMKs/s (RTT 3.0)
:~$

oclHashcat

oclHashcat looks good as well. One thing that's caught my attention is that HW Monitors 2-4 show 0% GPU. I'm not sure how to take that, but it's something I'll definitely need to investigate.

 
user@ubuntu:~$ sudo ./oclExample.sh
oclHashcat v0.26 by atom starting...

Digests: 6494 entries, 6494 unique
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes
Platform: AMD compatible platform found
Watchdog: Temperature limit set to 90c
Device #1: Cayman, 1024MB, 0Mhz, 22MCU
Device #2: Cayman, 1024MB, 0Mhz, 22MCU
Device #3: Cayman, 1024MB, 0Mhz, 22MCU
Device #4: Cayman, 1024MB, 0Mhz, 22MCU
NOTE: gpu-accel auto-adjusted to: 16
Device #1: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #2: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #3: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #4: Kernel ./kernels/4098/m0000.Cayman.64.kernel
42cceb8a0d3ca82b8fb6831f38e52a11:mega00000000
7becb9424f38abff581f6f2a82ff436a:sail00

--- CUT ----

c7d956020b614de2dd7772d22cdd459f:palytandra
[s]tatus [p]ause [r]esume [q]uit =>
Status.......: Exhausted
Hash.Type....: MD5
Mode.Left....: Mask '?l?l?l?l' (456976)
Mode.Right...: Dict 'example.dict' (129988)
Speed.GPU*...: 1761.3M/s
Recovered....: 1359/6494 Digests, 0/1 Salts
Progress.....: 59401396288/59401396288 (100.00%)
Running......: 18 secs
Estimated....: 0 secs
HW.Monitor.#1: 74% GPU, 71c Temp
HW.Monitor.#2:  0% GPU, 53c Temp
HW.Monitor.#3:  0% GPU, 43c Temp
HW.Monitor.#4:  0% GPU, 42c Temp

Started: Tue Nov 22 20:25:38 2011
Stopped: Tue Nov 22 20:25:56 2011

john

john only uses the CPUs, so no spectacular GPU stats here, but still some solid results.

john-1.7.8-jumbo-7/run$ ./john --test=30
Benchmarking: Traditional DES [128/128 BS SSE2-16]... DONE
Many salts: 4670K c/s real, 4685K c/s virtual
Only one salt: 3908K c/s real, 3920K c/s virtual

Benchmarking: BSDI DES (x725) [128/128 BS SSE2-16]... DONE
Many salts: 168806 c/s real, 169314 c/s virtual
Only one salt: 163686 c/s real, 164178 c/s virtual

Benchmarking: FreeBSD MD5 [8x]... DONE
Raw: 9768 c/s real, 9798 c/s virtual

Benchmarking: OpenBSD Blowfish (x32) [32/64 X2]... DONE
Raw: 870 c/s real, 873 c/s virtual

Benchmarking: Kerberos AFS DES [48/64 4K]... DONE
Short: 478762 c/s real, 480203 c/s virtual
Long: 1582K c/s real, 1587K c/s virtual

Benchmarking: LM DES [128/128 BS SSE2-16]... DONE
Raw: 25261K c/s real, 25337K c/s virtual

Benchmarking: Eggdrop [blowfish]... DONE
Raw: 29104 c/s real, 29191 c/s virtual

Benchmarking: DIGEST-MD5 [DIGEST-MD5 authentication]... DONE
Many salts: 1870K c/s real, 1876K c/s virtual
Only one salt: 1871K c/s real, 1876K c/s virtual

Benchmarking: More Secure Internet Password [RSA MD defined by BSAFE 1.x - Lotus v6]... DONE
Many salts: 101443 c/s real, 101748 c/s virtual
Only one salt: 60196 c/s real, 60378 c/s virtual

Benchmarking: EPiServer SID Hashes [SHA-1]... DONE
Many salts: 5542K c/s real, 5560K c/s virtual
Only one salt: 5257K c/s real, 5273K c/s virtual

Benchmarking: HTTP Digest access authentication [HDAA-MD5]... DONE
Many salts: 1988K c/s real, 1995K c/s virtual
Only one salt: 1940K c/s real, 1945K c/s virtual

Benchmarking: IPB2 MD5 [Invision Power Board 2.x salted MD5]... DONE
Many salts: 3988K c/s real, 4000K c/s virtual
Only one salt: 2485K c/s real, 2492K c/s virtual

Benchmarking: Kerberos v4 TGT [krb4 DES]... DONE
Raw: 3601K c/s real, 3612K c/s virtual

Benchmarking: Kerberos v5 TGT [krb5 3DES (des3-cbc-sha1)]... DONE
Raw: 63938 c/s real, 64130 c/s virtual

Benchmarking: MSCHAPv2 C/R MD4 DES [mschapv2]... DONE
Many salts: 3654K c/s real, 3666K c/s virtual
Only one salt: 2632K c/s real, 2640K c/s virtual

Benchmarking: MYSQL_fast [mysql-fast]... DONE
Raw: 42879K c/s real, 43009K c/s virtual

Benchmarking: MYSQL [mysql]... DONE
Raw: 5170K c/s real, 5185K c/s virtual

Benchmarking: LM C/R DES [netlm]... DONE
Many salts: 3621K c/s real, 3632K c/s virtual
Only one salt: 1216K c/s real, 1220K c/s virtual

Benchmarking: LMv2 C/R MD4 HMAC-MD5 [netlmv2]... DONE
Many salts: 1124K c/s real, 1127K c/s virtual
Only one salt: 908930 c/s real, 911665 c/s virtual

Benchmarking: NTLMv1 C/R MD4 DES [ESS MD5] [netntlm]... DONE
Many salts: 3651K c/s real, 3664K c/s virtual
Only one salt: 2652K c/s real, 2660K c/s virtual

Benchmarking: NTLMv2 C/R MD4 HMAC-MD5 [netntlmv2]... DONE
Many salts: 985779 c/s real, 989076 c/s virtual
Only one salt: 821367 c/s real, 823563 c/s virtual

Benchmarking: HalfLM C/R DES [nethalflm]... DONE
Many salts: 3622K c/s real, 3633K c/s virtual
Only one salt: 1825K c/s real, 1831K c/s virtual

Benchmarking: Netscape LDAP SSHA [salted SHA-1]... DONE
Many salts: 5197K c/s real, 5215K c/s virtual
Only one salt: 4738K c/s real, 4752K c/s virtual

Benchmarking: Netscape LDAP SHA [SHA-1]... DONE
Raw: 5029K c/s real, 5046K c/s virtual

Benchmarking: Netscreen MD5 [NS MD5]... DONE
Raw: 6159K c/s real, 6180K c/s virtual

Benchmarking: NT MD4 [128/128 X2 SSE2-16]... DONE
Raw: 47690K c/s real, 47849K c/s virtual

Benchmarking: OpenLDAP SSHA [salted SHA-1]... DONE
Many salts: 5281K c/s real, 5299K c/s virtual
Only one salt: 4883K c/s real, 4896K c/s virtual

Benchmarking: PHPS -- md5(md5($pass).$salt) [SSE2 16x4x2 (intr)]... DONE
Many salts: 9277K c/s real, 9305K c/s virtual
Only one salt: 3937K c/s real, 3949K c/s virtual

Benchmarking: Post.Office MD5 [STD]... DONE
Many salts: 4450K c/s real, 4463K c/s virtual
Only one salt: 4135K c/s real, 4146K c/s virtual

Benchmarking: Mac OS X 10.4 - 10.6 salted SHA-1 [32/64]... DONE
Many salts: 5886K c/s real, 5903K c/s virtual
Only one salt: 5575K c/s real, 5591K c/s virtual

Benchmarking: HMAC MD5 [hmac-md5]... DONE
Raw: 2043K c/s real, 2049K c/s virtual

Benchmarking: Lotus5 [Lotus v5 Proprietary]... DONE
Raw: 202395 c/s real, 203004 c/s virtual

Benchmarking: Generic salted MD4 [32/64]... DONE
Many salts: 9250K c/s real, 9278K c/s virtual
Only one salt: 8647K c/s real, 8673K c/s virtual

Benchmarking: MediaWiki -- md5($s.'-'.md5($p)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8465K c/s real, 8490K c/s virtual
Only one salt: 3815K c/s real, 3827K c/s virtual

Benchmarking: M$ Cache Hash [Generic 1x]... DONE
Many salts: 21450K c/s real, 21522K c/s virtual
Only one salt: 8711K c/s real, 8734K c/s virtual

Benchmarking: M$ Cache Hash 2 (DCC2) [SSE2-para 8x]... DONE
Raw: 402 c/s real, 403 c/s virtual

Benchmarking: MS Kerberos 5 AS-REQ Pre-Auth [mskrb5]... DONE
Many salts: 1153K c/s real, 1157K c/s virtual
Only one salt: 634491 c/s real, 636400 c/s virtual

Benchmarking: MS-SQL [mssql]... DONE
Many salts: 5505K c/s real, 5523K c/s virtual
Only one salt: 4844K c/s real, 4858K c/s virtual

Benchmarking: MS-SQL05 [ms-sql05]... DONE
Many salts: 5502K c/s real, 5521K c/s virtual
Only one salt: 5141K c/s real, 5156K c/s virtual

Benchmarking: MySQL 4.1 double-SHA-1 [mysql-sha1]... DONE
Raw: 2713K c/s real, 2722K c/s virtual

Benchmarking: Oracle 11g [oracle11]... DONE
Many salts: 5270K c/s real, 5287K c/s virtual
Only one salt: 4746K c/s real, 4759K c/s virtual

Benchmarking: Oracle [oracle]... DONE
Raw: 929803 c/s real, 932601 c/s virtual

Benchmarking: PHPass MD5 [SSE2 2x4x2 (intr)]... DONE
Raw: 4891 c/s real, 4905 c/s virtual

Benchmarking: PIX MD5 [pix-md5]... DONE
Raw: 6940K c/s real, 6963K c/s virtual

Benchmarking: pkzip [N/A]... DONE
Many salts: 10279K c/s real, 10314K c/s virtual
Only one salt: 7708K c/s real, 7731K c/s virtual

Benchmarking: Raw MD4 [32/64]... DONE
Raw: 9177K c/s real, 9205K c/s virtual

Benchmarking: Raw MD5 [SSE2 16x4x2 (intr)]... DONE
Raw: 8754K c/s real, 8780K c/s virtual

Benchmarking: Raw SHA-1(8x) [SHA-1]... DONE
Raw: 7107K c/s real, 7128K c/s virtual

Benchmarking: md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7508K c/s real, 7531K c/s virtual

Benchmarking: Salted SHA(8x) [SHA-1]... DONE
Many salts: 7616K c/s real, 7638K c/s virtual
Only one salt: 6608K c/s real, 6628K c/s virtual

Benchmarking: SAP BCODE [sapb]... DONE
Many salts: 2387K c/s real, 2394K c/s virtual
Only one salt: 2047K c/s real, 2054K c/s virtual

Benchmarking: SAP CODVN G (PASSCODE) [sapg]... DONE
Many salts: 1753K c/s real, 1758K c/s virtual
Only one salt: 1647K c/s real, 1651K c/s virtual

Benchmarking: Generic salted SHA-1 [32/64]... DONE
Many salts: 5175K c/s real, 5191K c/s virtual
Only one salt: 4947K c/s real, 4962K c/s virtual

Benchmarking: Tripcode DES [48/64 4K]... DONE
Raw: 383812 c/s real, 384967 c/s virtual

Benchmarking:  md5_gen(0): md5($p)  (raw-md5)  [SSE2 16x4x2 (intr)]... DONE
Raw: 8783K c/s real, 8809K c/s virtual

Benchmarking:  md5_gen(1): md5($p.$s)  (joomla)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6881K c/s real, 6901K c/s virtual

Benchmarking:  md5_gen(2): md5(md5($p))  (e107)  [SSE2 16x4x2 (intr)]... DONE
Raw: 4504K c/s real, 4517K c/s virtual

Benchmarking:  md5_gen(3): md5(md5(md5($p)))  [SSE2 16x4x2 (intr)]... DONE
Raw: 3028K c/s real, 3037K c/s virtual

Benchmarking:  md5_gen(4): md5($s.$p)  (OSC)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8218K c/s real, 8246K c/s virtual
Only one salt: 6975K c/s real, 6996K c/s virtual

Benchmarking:  md5_gen(5): md5($s.$p.$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 7621K c/s real, 7647K c/s virtual
Only one salt: 6508K c/s real, 6527K c/s virtual

Benchmarking:  md5_gen(6): md5(md5($p).$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 9179K c/s real, 9210K c/s virtual
Only one salt: 3980K c/s real, 3992K c/s virtual

Benchmarking:  md5_gen(8): md5(md5($s).$p)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8335K c/s real, 8363K c/s virtual
Only one salt: 7023K c/s real, 7044K c/s virtual

Benchmarking:  md5_gen(9): md5($s.md5($p))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 8166K c/s real, 8193K c/s virtual
Only one salt: 3772K c/s real, 3783K c/s virtual

Benchmarking:  md5_gen(10): md5($s.md5($s.$p))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4129K c/s real, 4143K c/s virtual
Only one salt: 3774K c/s real, 3786K c/s virtual

Benchmarking:  md5_gen(11): md5($s.md5($p.$s))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4150K c/s real, 4164K c/s virtual
Only one salt: 3798K c/s real, 3809K c/s virtual

Benchmarking:  md5_gen(12): md5(md5($s).md5($p))  (IPB)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4408K c/s real, 4423K c/s virtual
Only one salt: 2623K c/s real, 2631K c/s virtual

Benchmarking:  md5_gen(13): md5(md5($p).md5($s))  [SSE2 16x4x2 (intr)]... DONE
Many salts: 4404K c/s real, 4419K c/s virtual
Only one salt: 2626K c/s real, 2634K c/s virtual

Benchmarking:  md5_gen(14): md5($s.md5($p).$s)  [SSE2 16x4x2 (intr)]... DONE
Many salts: 7626K c/s real, 7649K c/s virtual
Only one salt: 3795K c/s real, 3806K c/s virtual

Benchmarking:  md5_gen(15): md5($u.md5($p).$s)  [64x2 (MD5_Body)]... DONE
Many salts: 9853K c/s real, 9883K c/s virtual
Only one salt: 4757K c/s real, 4772K c/s virtual

Benchmarking:  md5_gen(16): md5(md5(md5($p).$s).$s2)  [64x2 (MD5_Body)]... DONE
Many salts: 5344K c/s real, 5362K c/s virtual
Only one salt: 3170K c/s real, 3179K c/s virtual

Benchmarking:  md5_gen(17): phpass ($P$ or $H$)  [SSE2 2x4x2 (intr)]... DONE
Raw: 4902 c/s real, 4915 c/s virtual

Benchmarking:  md5_gen(18): md5($s.Y.$p.0xF7.$s) (Post.Office MD5)  [64x2 (MD5_Body)]... DONE
Many salts: 4341K c/s real, 4354K c/s virtual
Only one salt: 3981K c/s real, 3993K c/s virtual

Benchmarking:  md5_gen(19): Cisco PIX (MD5) [SSE2 16x4x2 (intr)]... DONE
Raw: 7481K c/s real, 7503K c/s virtual

Benchmarking:  md5_gen(20): Cisco PIX (MD5 salted) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8171K c/s real, 8196K c/s virtual
Only one salt: 6907K c/s real, 6928K c/s virtual

Benchmarking:  md5_gen(21): HTTP Digest Access Auth [SSE2 16x4x2 (intr)]... DONE
Many salts: 2027K c/s real, 2033K c/s virtual
Only one salt: 1960K c/s real, 1966K c/s virtual

Benchmarking:  md5_gen(22): md5(sha1($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3391K c/s real, 3401K c/s virtual

Benchmarking:  md5_gen(23): sha1(md5($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3223K c/s real, 3233K c/s virtual

Benchmarking:  md5_gen(24): sha1($p.$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 6076K c/s real, 6094K c/s virtual
Only one salt: 5242K c/s real, 5258K c/s virtual

Benchmarking:  md5_gen(25): sha1($s.$p) [SSE2 16x4x2 (intr)]... DONE
Many salts: 5946K c/s real, 5964K c/s virtual
Only one salt: 5315K c/s real, 5331K c/s virtual

Benchmarking:  md5_gen(26): sha1($p) raw-sha1 [4x2]... DONE
Raw: 6373K c/s real, 6393K c/s virtual

Benchmarking:  md5_gen(27): FreeBSD MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9502 c/s real, 9531 c/s virtual

Benchmarking:  md5_gen(28): Apache MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9456 c/s real, 9485 c/s virtual

Benchmarking:  md5_gen(29): md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7411K c/s real, 7436K c/s virtual

Benchmarking: md5_gen(1001) md5(md5(md5(md5($p)))) [SSE2 16x4x2 (intr)]... DONE
Raw: 2270K c/s real, 2278K c/s virtual

Benchmarking: md5_gen(1002) md5(md5(md5(md5(md5($p))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1823K c/s real, 1829K c/s virtual

Benchmarking: md5_gen(1003) md5(md5($p).md5($p)) [64x2 (MD5_Body)]... DONE
Raw: 2896K c/s real, 2905K c/s virtual

Benchmarking: md5_gen(1004) md5(md5(md5(md5(md5(md5($p)))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1523K c/s real, 1528K c/s virtual

Benchmarking: md5_gen(1005) md5(md5(md5(md5(md5(md5(md5($p))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1308K c/s real, 1312K c/s virtual

Benchmarking: md5_gen(1006) md5(md5(md5(md5(md5(md5(md5(md5($p)))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1145K c/s real, 1149K c/s virtual

Benchmarking: md5_gen(1008) md5($p.$s) [joomla] [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6868K c/s real, 6889K c/s virtual

Benchmarking: Raw SHA-224 [32/64]... DONE
Raw: 2577K c/s real, 2584K c/s virtual

Benchmarking: Raw SHA-256 [32/64]... DONE
Raw: 2600K c/s real, 2608K c/s virtual

Benchmarking: Raw SHA-384 [64/64]... DONE
Raw: 2054K c/s real, 2061K c/s virtual

Benchmarking: Raw SHA-512 [64/64]... DONE
Raw: 2054K c/s real, 2060K c/s virtual

Benchmarking: Mac OS X 10.7+ salted SHA-512 [64/64]... DONE
Many salts: 2098K c/s real, 2104K c/s virtual
Only one salt: 2056K c/s real, 2062K c/s virtual

Benchmarking: hmailserver [32/64]... DONE
Many salts: 2607K c/s real, 2614K c/s virtual
Only one salt: 2488K c/s real, 2495K c/s virtual

Benchmarking: sybasease [32/64]... DONE
Many salts: 344041 c/s real, 345076 c/s virtual
Only one salt: 341478 c/s real, 342505 c/s virtual

Benchmarking: generic crypt(3) [?/64]... DONE
Many salts: 405132 c/s real, 406351 c/s virtual
Only one salt: 403443 c/s real, 404657 c/s virtual

Benchmarking: ssh [32/64]... DONE
Raw: 268979 c/s real, 269878 c/s virtual

Benchmarking: pdf [32/64]... DONE
Many salts: 22014 c/s real, 22087 c/s virtual
Only one salt: 45401 c/s real, 45537 c/s virtual

Benchmarking: rar [32/64]... DONE
Raw: 48.2 c/s real, 48.3 c/s virtual

Benchmarking: zip [32/64]... DONE
Raw: 766 c/s real, 769 c/s virtual

Benchmarking: dummy [N/A]... DONE
Raw: 126288K c/s real, 126710K c/s virtual
    

Wrap up

Like many installations, we ran into a couple of road bumps starting out, but now our server is looking good. Before we make it live, we'll lock down the user permissions, roles, and set up a few cron jobs to keep the system tidy.

What do you think? Got any tips for our rig? Have a few pointers from your own experiences? Drop us some info in the comments below.