We do a ton of password cracking for our clients. Sometimes its because we've
compromised one system and are looking for password reuse, sometimes its because we're pulling password complexity statistics to prove a point, and sometimes we're just trying to break a WPA-PSK. Nonetheless it helps to have a beefy system on your side. We've recently revamped our cracking server, here is some information about its specifications and configuration.
Hardware Specs
Total: $2,629.78 USD
Installation
Hardware Mods
Although there was a slot available on the motherboard, the 4th video card really didn't fit in with the others (Video card bullying), so we made the decision to mount it in a sort of "unique" way. I'm sure the air-flow/cooling Gods aren't happy with this decision, but it was the only option. Using a flexible PCIe extender, we mounted the card (cringe).PCI Express Flexible Extender |  |
You can see in the picture to the right, we dremel'ed out some holes in the back of the case to accommodate the connectors on the video card, and to provide some stability. | |
 |  |
Hard drive/RAID Configuration
The 3TB drives required us to leverage Grub2 and GPT. Also it was important that we back up this system as best as we could in the case of hardware failure so we wanted to keep it simple with RAID1.Intel FakeRAID via the BIOS - FAIL
Initially we wanted to use RAID 1 via the Intel RAID functionality built into the motherboard but it didn't seem like it was fully supported by dmraid or Ubuntu Server 10.04.3. For whatever reason it would only show up as a 801GB drive in parted and the installation wouldn't allow any partition to be created outside of that 801GB. So we abandoned it.More fail info here: http://ubuntuforums.org/showthread.php?p=11477517
Software RAID (mdraid/mdadm) for the entire disk - FAIL
We wanted the drive to be fully redundant so we attempted to create a pretty large root partition and just a swap partition as linux_raid. The Ubuntu installation went fine with this attempt but the grub install failed. After a bunch of messing around we couldn't get grub properly installed. So we took a more traditional method of partitioning.More fail info here: http://ubuntuforums.org/showthread.php?p=11477507
Software RAID (mdraid/mdadm) for root partition - WIN!
Here's how we ended up partitioning both (/dev/sda, /dev/sdb) drives. We did this before installation by booting to a LiveCD. root@ubuntu:~# parted /dev/sda mklabel gpt
root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart grub 34 4130
root@ubuntu:~# parted -a cyl /dev/sdb set 1 bios_grub on
root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart boot 4131 1052706
root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart primary 1052707 5852144526
root@ubuntu:~# parted -a cyl /dev/sdb set 3 raid on
root@ubuntu:~# parted -a cyl /dev/sda unit s mkpart swap 5852144527s 5860533134s
root@ubuntu:~# parted /dev/sda unit compact print free
Model: ATA Hitachi HDS72303 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 17.4kB 2115kB 2098kB grub bios_grub
2 2115kB 539MB 537MB ext2 boot
3 539MB 2996GB 2996GB xfs primary raid
4 2996GB 3001GB 4295MB linux-swap(v1) swap
root@ubuntu:~# cat /etc/mdadm/mdadm.conf | grep ARRAY
ARRAY /dev/md0 level=raid1 metadata=1.0 num-devices=2 UUID=43dd56be:498abe6a:4421ece1:0659a71e name=0
Once the installation finished, grub2 still didn't immediately boot, so we booted off of the LiveCD again and installed it manually:
root@ubuntu:~# mount -t xfs /dev/sda3 /mnt
root@ubuntu:~# cp /mnt/etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf
root@ubuntu:~# mdadm --assemble /dev/md0
mdadm: /dev/md0 has been started with 1 drive (out of 2).
root@ubuntu:~# umount /mnt
root@ubuntu:~# mount /dev/md0 /mnt
root@ubuntu:~# mount /dev/sda2 /mnt/boot
root@ubuntu:~# for i in /dev /dev/pts /proc /sys; do sudo mount -B $i /mnt$i; done
root@ubuntu:~# chroot /mnt
root@ubuntu:/# grub-mkdevicemap
root@ubuntu:/# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-35-generic
Found initrd image: /boot/initrd.img-2.6.32-35-generic
Found memtest86+ image: /boot/memtest86+.bin
root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sda
Installation finished. No error reported.
root@ubuntu:/# grub-install --no-floppy --recheck --modules='biosdisk ext2' /dev/sdb
Installation finished. No error reported.
Installing OpenCL and AMD Drivers
Next we'll need to set up all those graphics cards. Since its AMD cards, we'll install the AMD OpenCL SDK and the AMD driver.user@ubuntu:~$ mkdir AMD-APP-SDK-v2.5-lnx64
user@ubuntu:~$ cd AMD-APP-SDK-v2.5-lnx64/
user@ubuntu:~$ wget http://developer.amd.com/Downloads/AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ tar -zxvf AMD-APP-SDK-v2.5-lnx64.tgz
user@ubuntu:~$ sudo ./Install-AMD-APP.sh
user@ubuntu:~$ cd /opt/AMDAPP/samples/opencl/bin/x86_64
user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ ./HelloCL
HelloCL!
Getting Platform Information
No protocol specified
Creating a context AMD platform
Getting device info
Loading and compiling CL source
Running CL program
Done
Passed!
user@ubuntu:/opt/AMDAPP/samples/opencl/bin/x86_64$ cd ~/
user@ubuntu:~$ sudo apt-get install libssl-dev build-essential python-dev libpcap-dev
user@ubuntu:~$ export AMDAPPSDKROOT="/opt/AMDAPP"
user@ubuntu:~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/opt/AMDAPP/lib/x86_64":"/opt/AMDAPP/lib/x86"
user@ubuntu:~$ export ATISTREAMSDKROOT=$AMDAPPSDKROOT
user@ubuntu:~$ chmod +x ati-driver-installer-11-11-x86.x86_64.run
user@ubuntu:~$ sudo ./ati-driver-installer-11-11-x86.x86_64.run
Installing Pyrit with OpenCL support
Pyrit is pretty awesome, but it doesn't take into account the latest AMD SDK. If you've been following the installation thus far, you should be ok. However you'll see that we'll have to make a couple changes in order to get everything working ok.user@ubuntu:~$ wget http://pyrit.googlecode.com/files/cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf cpyrit-opencl-0.4.0.tar.gz
user@ubuntu:~$ cd cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ vi setup.py
'/opt/AMDAPP/include', \
user@ubuntu:~/cpyrit-opencl-0.4.0$ python setup.py build
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install
user@ubuntu:~/cpyrit-opencl-0.4.0$ cd ..
user@ubuntu:~$ wget http://pyrit.googlecode.com/files/pyrit-0.4.0.tar.gz
user@ubuntu:~$ tar -zxvf pyrit-0.4.0.tar.gz
user@ubuntu:~$ cd pyrit-0.4.0
user@ubuntu:~/pyrit-0.4.0$ python setup.py build
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install
user@ubuntu:~/pyrit-0.4.0$ sudo pyrit list_cores
user@ubuntu:~$ sudo service gdm stop
user@ubuntu:~$ sudo aticonfig --adapter=all --initial
user@ubuntu:~$ sudo service gdm start
user@ubuntu:~$ sudo pyrit benchmark
Pyrit 0.4.0 (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+
Running benchmark (151933.5 PMKs/s)... /
Computed 151933.49 PMKs/s total.
#1: 'OpenCL-Device 'Cayman'': 41098.6 PMKs/s (RTT 2.7)
#2: 'OpenCL-Device 'Cayman'': 38496.3 PMKs/s (RTT 2.6)
#3: 'OpenCL-Device 'Cayman'': 42851.1 PMKs/s (RTT 2.6)
#4: 'OpenCL-Device 'Cayman'': 43023.6 PMKs/s (RTT 2.7)
#5: 'CPU-Core (SSE2)': 635.3 PMKs/s (RTT 3.5)
#6: 'CPU-Core (SSE2)': 588.7 PMKs/s (RTT 3.3)
#7: 'CPU-Core (SSE2)': 879.4 PMKs/s (RTT 2.8)
#8: 'CPU-Core (SSE2)': 961.1 PMKs/s (RTT 2.8)
Installing pyrit with CAL++ support
While 151933.5 PMKs/s is freaking awesome, we know that this isn't as fast as it *should* be. So instead of using the stable version of pyrit (0.4.0) with OpenCL, we decided to go with the SVN version of pyrit (0.4.1-dev) using CAL++.The first thing we'll have to do is uninstall pyrit (trust me, its worth it):
user@ubuntu:~$ cd pyrit-0.4.0/
user@ubuntu:~/pyrit-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)
user@ubuntu:~pyrit-0.4.0$ cd ../cpyrit-opencl-0.4.0/
user@ubuntu:~/cpyrit-opencl-0.4.0$ sudo python setup.py install --record files.txt; sudo rm $(cat files.txt)
user@ubuntu:~$ sudo apt-get install libboost-all-dev cmake
user@ubuntu:~$ tar -zxvf calpp-0.90.tar.gz
user@ubuntu:~$ cd calpp-0.90
FIND_LIBRARY( LIB_ATICALCL aticalcl PATHS "$ENV{ATISTREAMSDKROOT}/lib" )
FIND_LIBRARY( LIB_ATICALRT aticalrt PATHS "$ENV{ATISTREAMSDKROOT}/lib" )
FIND_PATH( LIB_ATICAL_INCLUDE NAMES cal.h calcl.h PATHS "$ENV{ATISTREAMSDKROOT}/include" )
FIND_LIBRARY( LIB_ATICALCL aticalcl PATHS "/opt/AMDAPP/lib" )
FIND_LIBRARY( LIB_ATICALRT aticalrt PATHS "/opt/AMDAPP/lib" )
FIND_PATH( LIB_ATICAL_INCLUDE NAMES cal.h calcl.h PATHS "/opt/AMDAPP/include/CAL" )
user@ubuntu:~/calpp-0.90$ sudo make install
user@ubuntu:~$ sudo apt-get install subversion
user@ubuntu:~$ svn checkout http://pyrit.googlecode.com/svn/trunk/ pyrit_svn
user@ubuntu:~$ cd pyrit_svn/cpyrit_calpp/
edit setup.py line 28 to read:
VERSION = '0.4.1-dev'
CALPP_INC_DIR = '/opt/AMDAPP'
CALPP_INC_DIRS.append(os.path.join(CALPP_INC_DIR, 'include/CAL'))
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ python setup.py build
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ sudo python setup.py install
user@ubuntu:~/pyrit_svn/cpyrit_calpp$ cd ../pyrit/
user@ubuntu:~/pyrit_svn/pyrit$ python setup.py build
user@ubuntu:~/pyrit_svn/pyritsudo python setup.py install
oclHashcat
The oclHashcat install is extremely easy. Just decompress.user@ubuntu:~$ sudo apt-get install p7zip
user@ubuntu:~$ p7zip -d oclHashcat-0.26.7z
john
user@ubuntu:~$ wget http://www.openwall.com/john/g/john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ tar -jxvf john-1.7.8-jumbo-8.tar.bz2
user@ubuntu:~$ cd john-1.7.8-jumbo-8/src/
user@ubuntu:~/john-1.7.8-jumbo-8/src$ make linux-x86-64
Next, we downloaded KoreLogic's Defcon 10 rulesets and added them to our john.conf
user@ubuntu:~/john-1.7.8-jumbo-8/src$ cd ../run
user@ubuntu:~/john-1.7.8-jumbo-8/run$ wget http://contest-2010.korelogic.com/rules.txt
user@ubuntu:~/john-1.7.8-jumbo-8/run$ cat rules.txt >> john.conf
rcracki
Newer rainbow tables are .RTI files, meaning they're indexed rainbow tables, which are supposed to be faster then the normal ones. Since we have a handful of those, we'll need to install rcracki.user@ubuntu:~$ wget http://www.freerainbowtables.com/download/rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ p7zip -d rcracki_mt_0.6.6_src.7z
user@ubuntu:~$ cd rcracki_mt_0.6.6_src/
Edit the Makefile, and change the OPTIMIZATION variable to read:
OPTIMIZATION = -O3 -mtune=native
Now we can compile
user@ubuntu:~/rcracki_mt_0.6.6_src$ make
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo make install
user@ubuntu:~/rcracki_mt_0.6.6_src$ cd ..
user@ubuntu:~$ sudo mv rcracki_mt_0.6.6_src /cracking/bin/
user@ubuntu:~$ cd /cracking/bin/rcracki_mt_0.6.6_src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mkdir src
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo mv BaseRTReader.* Cha* COPYING CrackEngine.* fast_md* global.h Hash* INSTALLING.txt lm2ntlm.* Makefile MemoryPool.* Public.* RainbowCrack.* rcrackiThread.* README.txt RT* sha1.* TODO src/
user@ubuntu:~/rcracki_mt_0.6.6_src$ sudo chmod o+w rcracki.*
rcrack
To support the older .RT rainbow tables, we can use standard rcrack. Version 1.5 comes pre-compiled, so the installation is really just a download and decompress.user@ubuntu:~$ wget http://project-rainbowcrack.com/rainbowcrack-1.5-linux64.zip
user@ubuntu:~$ unzip rainbowcrack-1.5-linux64.zip
Downloads
Rainbow Tables
What would a password cracking server, without a ton of Rainbow Tables. Luckily we have an entire of army of geeks at our disposal, so we didn't have to spend any time downloading. We copied over the following Rainbow Tables:fastlm_alpha-numeric#1-7_0
lm_all-space_1-7
lm_alpha_1-7
lm_alpha-numeric_1-7
lm_alpha-numeric-symbol32-space_1-7
ntlm-alpha-numeric-space_1-8
ntlm_alpha-space_1-9
ntlm_hybrid(loweralpha#6-6,numeric#1-3)
ntlm_hybrid(loweralpha#7-7,numeric#1-3)
ntlm_loweralpha-numeric-space_1-8
ntlm_loweralpha-numeric-symbol32-space_1-7_0
ntlm_loweralpha-space_1-9
ntlm_mixalpha-numeric_1-8
ntlm_mixalpha-numeric-all-space_1-6
ntlm_mixalpha-numeric-space_1-7_0
ntlm_numeric_1-12
Ophcrack_Tables-LM_Fast_Free
Ophcrack_Tables-NTLM_Free
sha1_loweralpha-numeric-space_1-8
wpa-h1kari_renderman
wpa-Offensive WPA Tables
Wordlists
Wordlists are one of those things that you can never seem to have enough of. We pulled together some popular ones, and I'm sure we'll be getting a lot of recommendations for others once the server reaches production.user@ubuntu:/cracking/wordlists$ wget http://download.openwall.net/pub/wordlists/all.gz
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/500-worst-passwords.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/twitter-banned.txt.bz2
user@ubuntu:/cracking/wordlists$ wget http://downloads.skullsecurity.org/passwords/english.txt.bz2
user@ubuntu:/cracking/wordlists$ bunzip2 *.bz2
user@ubuntu:/cracking/wordlists$ gunzip all.gz
Supporting a Multi-user environment
GPU Cracking
Since this system will be accessible by a bunch of people, I wanted to make sure no one stepped on eachother's toes. I wrote this simple wrapper so that if anyone tried to used oclHashcat or pyrit, while someone else is, they'd be told not to.user@ubuntu:~$ cat /cracking/bin/gpu-crack
#!/bin/bash
#
# gpu-crack - Simple GPU cracker wrapper for multi-user
# environments
#
# by Brad Antoniewicz
#
TMPFILE="/tmp/check.tmp"
ver="0.1"
SUDO_EXEC=/usr/bin/sudo
PYRIT_EXEC=/usr/local/bin/pyrit.real
HASHCAT_EXEC=/cracking/bin/oclhash/oclHashcat64.bin
isRunning=0;
echo -e "\n$0 v$ver"
echo "by brad a."
echo -e "-------------------------------------------------\n"
help() {
echo "Usage:"
echo -e "\t$0 [pyrit|hashcat] [options]\n"
echo "Define what program you want to crack with (pyrit or hashcat)"
echo "then provide the standard command line options that the cracker"
echo "supports."
}
checkProc() {
ps ax -o pid,user,etime,command | grep $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
if [[ $NUMPROC != 0 ]]; then
echo -e "[!] Found $NUMPROC instance of $1 running\n"
for ((i=1; i<=$NUMPROC; i++))
do
# awk NR==$i "$TMPFILE"
PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
echo -e "\tPID: $PID"
echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2`"
echo -e "\tRunning for: `awk NR==$i "$TMPFILE" | awk '{print $3}'`"
echo -e "\n"
isRunning=1;
done
else
echo -e "[-] No instances of $1 found"
fi
}
launchProc() {
count=0;
for x in "$@"
do
if [ $count != 0 ]; then
cmdlineArgs=$cmdlineArgs" "$x
fi
let count++
done
echo "[+] Launching $1 with the following options"
echo -e "\t $cmdlineArgs"
if [[ $1 == "pyrit" ]]; then
echo "[+] Running Pyrit"
$SUDO_EXEC $PYRIT_EXEC $cmdlineArgs
elif [[ $1 == "hashcat" ]]; then
echo "[+] Running oclHashcat"
$SUDO_EXEC $HASHCAT_EXEC $cmdlineArgs
fi
}
if [[ $# -ge 2 ]] && [[ $1 == "pyrit" || $1 == "hashcat" ]]; then
echo "[+] Checking for conflicting processes"
checkProc "pyrit"
checkProc "oclHash"
if [ $isRunning == 0 ]; then
launchProc $@
else
echo "[!!] Found conflicting process, contact owner and make it stop!"
fi
else
help
fi
General Access
Since rcrack, rcracki, and john sort of need to be run from their program directories, I created a couple of aliases to get the user into the right spot with ease. Also, we put most of these installs into a central location (/cracking/bin) so that we can maintain them easy (no real packages for these tools).user@ubuntu:~$ cat /etc/profile.d/pwserver.sh
#
# Set up all of the paths, directories, etc.. for password cracking
# by brad a.
#
alias john='cd /cracking/bin/john-bin; echo Changing directory - Please run john from here with ./john'
alias rcrack='cd /cracking/bin/rainbowcrack; echo -e "Changing directory - Please run rcrack from here with:\n./rcrack /path/to/tables/*.rt -f /path/to/pwdump"'
alias rcracki='cd /cracking/bin/rcracki_mt; echo -e "Changing directory - Please run rcracki from here with:\n./rcracki -f /path/to/pwdump /path/to/tables/"'
PATH=$PATH:/cracking/bin
export PATH
Changing MOTD
What would be a Foundstone server without a fun MOTD. I've omitted the actual welcome message (stored in /etc/motd.tail) as it might be a little bit offensive :), but the stats script that gets launched on login is below.user@ubuntu:~$ cd /etc/update-motd.d
user@ubuntu:/etc/update-motd.d$ sudo chmod -x *
user@ubuntu:/etc/update-motd.d$ sudo touch 10-pwserver
user@ubuntu:/etc/update-motd.d$ sudo chmod +x 10-pwserver
user@ubuntu:/etc/update-motd.d$ cat 10-pwserver
#!/bin/bash
#
#
# by Brad Antoniewicz
#
[ -f /etc/motd.tail ] && cat /etc/motd.tail || true
TMPFILE="/tmp/check.tmp"
ver="0.1"
isRunning=0;
processes="oclHashcat64.bin hashcat pyrit oclHash john rcrack rcracki rcracki_mt"
checkProc() {
ps ax -o pid,user,etime,command | grep -w $1 | grep -v grep | grep -v $0 | sed -e 's/^ //' > $TMPFILE
NUMPROC=`wc -l $TMPFILE | cut -d" " -f 1`
if [[ $NUMPROC != 0 ]]; then
for ((i=1; i<=$NUMPROC; i++))
do
# awk NR==$i "$TMPFILE"
PID=`awk NR==$i "$TMPFILE" | cut -d" " -f 1`
echo -e "\tUser: `awk NR==$i "$TMPFILE" | cut -d" " -f 2` - Process: $1 - PID: $PID - Running For: `awk NR==$i "$TMPFILE" | awk '{print $3}'`"
isRunning=1;
done
fi
}
echo Processes currently running
for i in $processes
do
checkProc $i
done
if [ $isRunning == 0 ]; then
echo -e "\tNone!"
fi
Benchmarks
pyrit
Our pyrit benchmarks are decently respectable:user@ubuntu:~$ gpu-crack pyrit benchmark
/usr/local/bin/gpu-crack v0.1by brad a.
-------------------------------------------------
[+] Checking for conflicting processes
[-] No instances of pyrit found
[-] No instances of oclHash found
[+] Launching pyrit with the following options
benchmark
[+] Running Pyrit
Pyrit 0.4.1-dev (svn r308) (C) 2008-2011 Lukas Lueg http://pyrit.googlecode.com
This code is distributed under the GNU General Public License v3+
Running benchmark (172644.8 PMKs/s)... /
Computed 172644.84 PMKs/s total.
#1: 'CAL++ Device #1 'AMD CAYMAN'': 51245.4 PMKs/s (RTT 1.0)
#2: 'CAL++ Device #2 'AMD CAYMAN'': 47993.0 PMKs/s (RTT 1.0)
#3: 'CAL++ Device #3 'AMD CAYMAN'': 51163.7 PMKs/s (RTT 1.0)
#4: 'CAL++ Device #4 'AMD CAYMAN'': 52829.2 PMKs/s (RTT 0.9)
#5: 'CPU-Core (SSE2/AES)': 713.9 PMKs/s (RTT 2.9)
#6: 'CPU-Core (SSE2/AES)': 736.3 PMKs/s (RTT 3.0)
#7: 'CPU-Core (SSE2/AES)': 821.7 PMKs/s (RTT 2.9)
#8: 'CPU-Core (SSE2/AES)': 763.8 PMKs/s (RTT 3.0)
:~$
oclHashcat
oclHashcat looks good as well. One thing that's caught my attention is that HW Monitors 2-4 show 0% GPU. I'm not sure how to take that, but it's something I'll definitely need to investigate.
user@ubuntu:~$ sudo ./oclExample.sh
oclHashcat v0.26 by atom starting...
Digests: 6494 entries, 6494 unique
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes
Platform: AMD compatible platform found
Watchdog: Temperature limit set to 90c
Device #1: Cayman, 1024MB, 0Mhz, 22MCU
Device #2: Cayman, 1024MB, 0Mhz, 22MCU
Device #3: Cayman, 1024MB, 0Mhz, 22MCU
Device #4: Cayman, 1024MB, 0Mhz, 22MCU
NOTE: gpu-accel auto-adjusted to: 16
Device #1: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #2: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #3: Kernel ./kernels/4098/m0000.Cayman.64.kernel
Device #4: Kernel ./kernels/4098/m0000.Cayman.64.kernel
42cceb8a0d3ca82b8fb6831f38e52a11:mega00000000
7becb9424f38abff581f6f2a82ff436a:sail00
--- CUT ----
c7d956020b614de2dd7772d22cdd459f:palytandra
[s]tatus [p]ause [r]esume [q]uit =>
Status.......: Exhausted
Hash.Type....: MD5
Mode.Left....: Mask '?l?l?l?l' (456976)
Mode.Right...: Dict 'example.dict' (129988)
Speed.GPU*...: 1761.3M/s
Recovered....: 1359/6494 Digests, 0/1 Salts
Progress.....: 59401396288/59401396288 (100.00%)
Running......: 18 secs
Estimated....: 0 secs
HW.Monitor.#1: 74% GPU, 71c Temp
HW.Monitor.#2: 0% GPU, 53c Temp
HW.Monitor.#3: 0% GPU, 43c Temp
HW.Monitor.#4: 0% GPU, 42c Temp
Started: Tue Nov 22 20:25:38 2011
Stopped: Tue Nov 22 20:25:56 2011
john
john only uses the CPUs, so no spectacular GPU stats here, but still some solid results.
john-1.7.8-jumbo-7/run$ ./john --test=30
Benchmarking: Traditional DES [128/128 BS SSE2-16]... DONE
Many salts: 4670K c/s real, 4685K c/s virtual
Only one salt: 3908K c/s real, 3920K c/s virtual
Benchmarking: BSDI DES (x725) [128/128 BS SSE2-16]... DONE
Many salts: 168806 c/s real, 169314 c/s virtual
Only one salt: 163686 c/s real, 164178 c/s virtual
Benchmarking: FreeBSD MD5 [8x]... DONE
Raw: 9768 c/s real, 9798 c/s virtual
Benchmarking: OpenBSD Blowfish (x32) [32/64 X2]... DONE
Raw: 870 c/s real, 873 c/s virtual
Benchmarking: Kerberos AFS DES [48/64 4K]... DONE
Short: 478762 c/s real, 480203 c/s virtual
Long: 1582K c/s real, 1587K c/s virtual
Benchmarking: LM DES [128/128 BS SSE2-16]... DONE
Raw: 25261K c/s real, 25337K c/s virtual
Benchmarking: Eggdrop [blowfish]... DONE
Raw: 29104 c/s real, 29191 c/s virtual
Benchmarking: DIGEST-MD5 [DIGEST-MD5 authentication]... DONE
Many salts: 1870K c/s real, 1876K c/s virtual
Only one salt: 1871K c/s real, 1876K c/s virtual
Benchmarking: More Secure Internet Password [RSA MD defined by BSAFE 1.x - Lotus v6]... DONE
Many salts: 101443 c/s real, 101748 c/s virtual
Only one salt: 60196 c/s real, 60378 c/s virtual
Benchmarking: EPiServer SID Hashes [SHA-1]... DONE
Many salts: 5542K c/s real, 5560K c/s virtual
Only one salt: 5257K c/s real, 5273K c/s virtual
Benchmarking: HTTP Digest access authentication [HDAA-MD5]... DONE
Many salts: 1988K c/s real, 1995K c/s virtual
Only one salt: 1940K c/s real, 1945K c/s virtual
Benchmarking: IPB2 MD5 [Invision Power Board 2.x salted MD5]... DONE
Many salts: 3988K c/s real, 4000K c/s virtual
Only one salt: 2485K c/s real, 2492K c/s virtual
Benchmarking: Kerberos v4 TGT [krb4 DES]... DONE
Raw: 3601K c/s real, 3612K c/s virtual
Benchmarking: Kerberos v5 TGT [krb5 3DES (des3-cbc-sha1)]... DONE
Raw: 63938 c/s real, 64130 c/s virtual
Benchmarking: MSCHAPv2 C/R MD4 DES [mschapv2]... DONE
Many salts: 3654K c/s real, 3666K c/s virtual
Only one salt: 2632K c/s real, 2640K c/s virtual
Benchmarking: MYSQL_fast [mysql-fast]... DONE
Raw: 42879K c/s real, 43009K c/s virtual
Benchmarking: MYSQL [mysql]... DONE
Raw: 5170K c/s real, 5185K c/s virtual
Benchmarking: LM C/R DES [netlm]... DONE
Many salts: 3621K c/s real, 3632K c/s virtual
Only one salt: 1216K c/s real, 1220K c/s virtual
Benchmarking: LMv2 C/R MD4 HMAC-MD5 [netlmv2]... DONE
Many salts: 1124K c/s real, 1127K c/s virtual
Only one salt: 908930 c/s real, 911665 c/s virtual
Benchmarking: NTLMv1 C/R MD4 DES [ESS MD5] [netntlm]... DONE
Many salts: 3651K c/s real, 3664K c/s virtual
Only one salt: 2652K c/s real, 2660K c/s virtual
Benchmarking: NTLMv2 C/R MD4 HMAC-MD5 [netntlmv2]... DONE
Many salts: 985779 c/s real, 989076 c/s virtual
Only one salt: 821367 c/s real, 823563 c/s virtual
Benchmarking: HalfLM C/R DES [nethalflm]... DONE
Many salts: 3622K c/s real, 3633K c/s virtual
Only one salt: 1825K c/s real, 1831K c/s virtual
Benchmarking: Netscape LDAP SSHA [salted SHA-1]... DONE
Many salts: 5197K c/s real, 5215K c/s virtual
Only one salt: 4738K c/s real, 4752K c/s virtual
Benchmarking: Netscape LDAP SHA [SHA-1]... DONE
Raw: 5029K c/s real, 5046K c/s virtual
Benchmarking: Netscreen MD5 [NS MD5]... DONE
Raw: 6159K c/s real, 6180K c/s virtual
Benchmarking: NT MD4 [128/128 X2 SSE2-16]... DONE
Raw: 47690K c/s real, 47849K c/s virtual
Benchmarking: OpenLDAP SSHA [salted SHA-1]... DONE
Many salts: 5281K c/s real, 5299K c/s virtual
Only one salt: 4883K c/s real, 4896K c/s virtual
Benchmarking: PHPS -- md5(md5($pass).$salt) [SSE2 16x4x2 (intr)]... DONE
Many salts: 9277K c/s real, 9305K c/s virtual
Only one salt: 3937K c/s real, 3949K c/s virtual
Benchmarking: Post.Office MD5 [STD]... DONE
Many salts: 4450K c/s real, 4463K c/s virtual
Only one salt: 4135K c/s real, 4146K c/s virtual
Benchmarking: Mac OS X 10.4 - 10.6 salted SHA-1 [32/64]... DONE
Many salts: 5886K c/s real, 5903K c/s virtual
Only one salt: 5575K c/s real, 5591K c/s virtual
Benchmarking: HMAC MD5 [hmac-md5]... DONE
Raw: 2043K c/s real, 2049K c/s virtual
Benchmarking: Lotus5 [Lotus v5 Proprietary]... DONE
Raw: 202395 c/s real, 203004 c/s virtual
Benchmarking: Generic salted MD4 [32/64]... DONE
Many salts: 9250K c/s real, 9278K c/s virtual
Only one salt: 8647K c/s real, 8673K c/s virtual
Benchmarking: MediaWiki -- md5($s.'-'.md5($p)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8465K c/s real, 8490K c/s virtual
Only one salt: 3815K c/s real, 3827K c/s virtual
Benchmarking: M$ Cache Hash [Generic 1x]... DONE
Many salts: 21450K c/s real, 21522K c/s virtual
Only one salt: 8711K c/s real, 8734K c/s virtual
Benchmarking: M$ Cache Hash 2 (DCC2) [SSE2-para 8x]... DONE
Raw: 402 c/s real, 403 c/s virtual
Benchmarking: MS Kerberos 5 AS-REQ Pre-Auth [mskrb5]... DONE
Many salts: 1153K c/s real, 1157K c/s virtual
Only one salt: 634491 c/s real, 636400 c/s virtual
Benchmarking: MS-SQL [mssql]... DONE
Many salts: 5505K c/s real, 5523K c/s virtual
Only one salt: 4844K c/s real, 4858K c/s virtual
Benchmarking: MS-SQL05 [ms-sql05]... DONE
Many salts: 5502K c/s real, 5521K c/s virtual
Only one salt: 5141K c/s real, 5156K c/s virtual
Benchmarking: MySQL 4.1 double-SHA-1 [mysql-sha1]... DONE
Raw: 2713K c/s real, 2722K c/s virtual
Benchmarking: Oracle 11g [oracle11]... DONE
Many salts: 5270K c/s real, 5287K c/s virtual
Only one salt: 4746K c/s real, 4759K c/s virtual
Benchmarking: Oracle [oracle]... DONE
Raw: 929803 c/s real, 932601 c/s virtual
Benchmarking: PHPass MD5 [SSE2 2x4x2 (intr)]... DONE
Raw: 4891 c/s real, 4905 c/s virtual
Benchmarking: PIX MD5 [pix-md5]... DONE
Raw: 6940K c/s real, 6963K c/s virtual
Benchmarking: pkzip [N/A]... DONE
Many salts: 10279K c/s real, 10314K c/s virtual
Only one salt: 7708K c/s real, 7731K c/s virtual
Benchmarking: Raw MD4 [32/64]... DONE
Raw: 9177K c/s real, 9205K c/s virtual
Benchmarking: Raw MD5 [SSE2 16x4x2 (intr)]... DONE
Raw: 8754K c/s real, 8780K c/s virtual
Benchmarking: Raw SHA-1(8x) [SHA-1]... DONE
Raw: 7107K c/s real, 7128K c/s virtual
Benchmarking: md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7508K c/s real, 7531K c/s virtual
Benchmarking: Salted SHA(8x) [SHA-1]... DONE
Many salts: 7616K c/s real, 7638K c/s virtual
Only one salt: 6608K c/s real, 6628K c/s virtual
Benchmarking: SAP BCODE [sapb]... DONE
Many salts: 2387K c/s real, 2394K c/s virtual
Only one salt: 2047K c/s real, 2054K c/s virtual
Benchmarking: SAP CODVN G (PASSCODE) [sapg]... DONE
Many salts: 1753K c/s real, 1758K c/s virtual
Only one salt: 1647K c/s real, 1651K c/s virtual
Benchmarking: Generic salted SHA-1 [32/64]... DONE
Many salts: 5175K c/s real, 5191K c/s virtual
Only one salt: 4947K c/s real, 4962K c/s virtual
Benchmarking: Tripcode DES [48/64 4K]... DONE
Raw: 383812 c/s real, 384967 c/s virtual
Benchmarking: md5_gen(0): md5($p) (raw-md5) [SSE2 16x4x2 (intr)]... DONE
Raw: 8783K c/s real, 8809K c/s virtual
Benchmarking: md5_gen(1): md5($p.$s) (joomla) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6881K c/s real, 6901K c/s virtual
Benchmarking: md5_gen(2): md5(md5($p)) (e107) [SSE2 16x4x2 (intr)]... DONE
Raw: 4504K c/s real, 4517K c/s virtual
Benchmarking: md5_gen(3): md5(md5(md5($p))) [SSE2 16x4x2 (intr)]... DONE
Raw: 3028K c/s real, 3037K c/s virtual
Benchmarking: md5_gen(4): md5($s.$p) (OSC) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8218K c/s real, 8246K c/s virtual
Only one salt: 6975K c/s real, 6996K c/s virtual
Benchmarking: md5_gen(5): md5($s.$p.$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 7621K c/s real, 7647K c/s virtual
Only one salt: 6508K c/s real, 6527K c/s virtual
Benchmarking: md5_gen(6): md5(md5($p).$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 9179K c/s real, 9210K c/s virtual
Only one salt: 3980K c/s real, 3992K c/s virtual
Benchmarking: md5_gen(8): md5(md5($s).$p) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8335K c/s real, 8363K c/s virtual
Only one salt: 7023K c/s real, 7044K c/s virtual
Benchmarking: md5_gen(9): md5($s.md5($p)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8166K c/s real, 8193K c/s virtual
Only one salt: 3772K c/s real, 3783K c/s virtual
Benchmarking: md5_gen(10): md5($s.md5($s.$p)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 4129K c/s real, 4143K c/s virtual
Only one salt: 3774K c/s real, 3786K c/s virtual
Benchmarking: md5_gen(11): md5($s.md5($p.$s)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 4150K c/s real, 4164K c/s virtual
Only one salt: 3798K c/s real, 3809K c/s virtual
Benchmarking: md5_gen(12): md5(md5($s).md5($p)) (IPB) [SSE2 16x4x2 (intr)]... DONE
Many salts: 4408K c/s real, 4423K c/s virtual
Only one salt: 2623K c/s real, 2631K c/s virtual
Benchmarking: md5_gen(13): md5(md5($p).md5($s)) [SSE2 16x4x2 (intr)]... DONE
Many salts: 4404K c/s real, 4419K c/s virtual
Only one salt: 2626K c/s real, 2634K c/s virtual
Benchmarking: md5_gen(14): md5($s.md5($p).$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 7626K c/s real, 7649K c/s virtual
Only one salt: 3795K c/s real, 3806K c/s virtual
Benchmarking: md5_gen(15): md5($u.md5($p).$s) [64x2 (MD5_Body)]... DONE
Many salts: 9853K c/s real, 9883K c/s virtual
Only one salt: 4757K c/s real, 4772K c/s virtual
Benchmarking: md5_gen(16): md5(md5(md5($p).$s).$s2) [64x2 (MD5_Body)]... DONE
Many salts: 5344K c/s real, 5362K c/s virtual
Only one salt: 3170K c/s real, 3179K c/s virtual
Benchmarking: md5_gen(17): phpass ($P$ or $H$) [SSE2 2x4x2 (intr)]... DONE
Raw: 4902 c/s real, 4915 c/s virtual
Benchmarking: md5_gen(18): md5($s.Y.$p.0xF7.$s) (Post.Office MD5) [64x2 (MD5_Body)]... DONE
Many salts: 4341K c/s real, 4354K c/s virtual
Only one salt: 3981K c/s real, 3993K c/s virtual
Benchmarking: md5_gen(19): Cisco PIX (MD5) [SSE2 16x4x2 (intr)]... DONE
Raw: 7481K c/s real, 7503K c/s virtual
Benchmarking: md5_gen(20): Cisco PIX (MD5 salted) [SSE2 16x4x2 (intr)]... DONE
Many salts: 8171K c/s real, 8196K c/s virtual
Only one salt: 6907K c/s real, 6928K c/s virtual
Benchmarking: md5_gen(21): HTTP Digest Access Auth [SSE2 16x4x2 (intr)]... DONE
Many salts: 2027K c/s real, 2033K c/s virtual
Only one salt: 1960K c/s real, 1966K c/s virtual
Benchmarking: md5_gen(22): md5(sha1($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3391K c/s real, 3401K c/s virtual
Benchmarking: md5_gen(23): sha1(md5($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 3223K c/s real, 3233K c/s virtual
Benchmarking: md5_gen(24): sha1($p.$s) [SSE2 16x4x2 (intr)]... DONE
Many salts: 6076K c/s real, 6094K c/s virtual
Only one salt: 5242K c/s real, 5258K c/s virtual
Benchmarking: md5_gen(25): sha1($s.$p) [SSE2 16x4x2 (intr)]... DONE
Many salts: 5946K c/s real, 5964K c/s virtual
Only one salt: 5315K c/s real, 5331K c/s virtual
Benchmarking: md5_gen(26): sha1($p) raw-sha1 [4x2]... DONE
Raw: 6373K c/s real, 6393K c/s virtual
Benchmarking: md5_gen(27): FreeBSD MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9502 c/s real, 9531 c/s virtual
Benchmarking: md5_gen(28): Apache MD5 [SSE2 4x2 (intr)]... DONE
Raw: 9456 c/s real, 9485 c/s virtual
Benchmarking: md5_gen(29): md5(unicode($p)) [SSE2 16x4x2 (intr)]... DONE
Raw: 7411K c/s real, 7436K c/s virtual
Benchmarking: md5_gen(1001) md5(md5(md5(md5($p)))) [SSE2 16x4x2 (intr)]... DONE
Raw: 2270K c/s real, 2278K c/s virtual
Benchmarking: md5_gen(1002) md5(md5(md5(md5(md5($p))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1823K c/s real, 1829K c/s virtual
Benchmarking: md5_gen(1003) md5(md5($p).md5($p)) [64x2 (MD5_Body)]... DONE
Raw: 2896K c/s real, 2905K c/s virtual
Benchmarking: md5_gen(1004) md5(md5(md5(md5(md5(md5($p)))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1523K c/s real, 1528K c/s virtual
Benchmarking: md5_gen(1005) md5(md5(md5(md5(md5(md5(md5($p))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1308K c/s real, 1312K c/s virtual
Benchmarking: md5_gen(1006) md5(md5(md5(md5(md5(md5(md5(md5($p)))))))) [SSE2 16x4x2 (intr)]... DONE
Raw: 1145K c/s real, 1149K c/s virtual
Benchmarking: md5_gen(1008) md5($p.$s) [joomla] [SSE2 16x4x2 (intr)]... DONE
Many salts: 8105K c/s real, 8130K c/s virtual
Only one salt: 6868K c/s real, 6889K c/s virtual
Benchmarking: Raw SHA-224 [32/64]... DONE
Raw: 2577K c/s real, 2584K c/s virtual
Benchmarking: Raw SHA-256 [32/64]... DONE
Raw: 2600K c/s real, 2608K c/s virtual
Benchmarking: Raw SHA-384 [64/64]... DONE
Raw: 2054K c/s real, 2061K c/s virtual
Benchmarking: Raw SHA-512 [64/64]... DONE
Raw: 2054K c/s real, 2060K c/s virtual
Benchmarking: Mac OS X 10.7+ salted SHA-512 [64/64]... DONE
Many salts: 2098K c/s real, 2104K c/s virtual
Only one salt: 2056K c/s real, 2062K c/s virtual
Benchmarking: hmailserver [32/64]... DONE
Many salts: 2607K c/s real, 2614K c/s virtual
Only one salt: 2488K c/s real, 2495K c/s virtual
Benchmarking: sybasease [32/64]... DONE
Many salts: 344041 c/s real, 345076 c/s virtual
Only one salt: 341478 c/s real, 342505 c/s virtual
Benchmarking: generic crypt(3) [?/64]... DONE
Many salts: 405132 c/s real, 406351 c/s virtual
Only one salt: 403443 c/s real, 404657 c/s virtual
Benchmarking: ssh [32/64]... DONE
Raw: 268979 c/s real, 269878 c/s virtual
Benchmarking: pdf [32/64]... DONE
Many salts: 22014 c/s real, 22087 c/s virtual
Only one salt: 45401 c/s real, 45537 c/s virtual
Benchmarking: rar [32/64]... DONE
Raw: 48.2 c/s real, 48.3 c/s virtual
Benchmarking: zip [32/64]... DONE
Raw: 766 c/s real, 769 c/s virtual
Benchmarking: dummy [N/A]... DONE
Raw: 126288K c/s real, 126710K c/s virtual
Wrap up
Like many installations, we ran into a couple of road bumps starting out, but now our server is looking good. Before we make it live, we'll lock down the user permissions, roles, and set up a few cron jobs to keep the system tidy.What do you think? Got any tips for our rig? Have a few pointers from your own experiences? Drop us some info in the comments below.
