GCC 4.6, LLVM/Clang 3.0, Open64 Benchmarks

[phoronix]

Phoronix: GCC 4.6, LLVM/Clang 3.0, Open64 Benchmarks

For those interested by last week's articles on the AMD Bulldozer benchmarks of the GCC, Open64, and LLVM/Clang compilers and the FX-8150 compiler tuning, but curious to know how the new LLVM/Clang 3.0 and other compilers perform on other processors, here's some new test results. These tests are looking at GCC 4.6, LLVM/Clang 3.0-RC1, and AMD Open64 compilers on Intel Sandy Bridge and AMD Shanghai hardware.

http://www.phoronix.com/vr.php?view=16665

[mcirsta]

What about that Ekopath compiler ? After all the fuss when it was open sourced I heard nothing about it. Maybe I missed something but still I'd like to see some results for that one too.

[Markore]

I would like to see comparison side-by-side with Solaris Studio (Formerly SunStudio).

[Cyborg16]

What I find most interesting is that Open64 performs inconsistently while consitently failing to compile on intel platforms.

Still a big gap between llvm and gcc in one test too.

[XorEaxEax]

I've often found the Phoronix test results very different from that of my own. The 7-zip test for instance, it yields very different results from what I get when I use the built in benchmark functionality so I figured I would do a real (as in non-synthetic) test using p7zip on my corei5 machine (nehalem):

GCC		-O1 -march=native		0m36.376s
GCC		-O1 -march=native		0m36.369s		Profile guided optimization
GCC4.7		-O1 -march=native		0m35.814s
GCC4.7		-O1 -march=native		0m35.761s		Profile guided optimization
Clang		-O1 -march=native		0m42.236s
GCC		-O2 -march=native		0m35.734s
GCC		-O2 -march=native		0m33.435s		Profile guided optimization
GCC4.7		-O2 -march=native		0m35.663s
GCC4.7		-O2 -march=native		0m33.831s		Profile guided optimization
Clang		-O2 -march=native		0m39.658s
GCC		-O3 -march=native		0m34.197s
GCC		-O3 -march=native		0m33.428s		Profile guided optimization
GCC4.7		-O3 -march=native		0m34.693s
GCC4.7		-O3 -march=native		0m33.224s		Profile guided optimization
Clang		-O3 -march=native		0m39.642s

These are the results of compressing an .iso file (Arch linux netinstall to be exact) on a core i5, 64bit Arch Linux system using the following compilers:
GCC 4.6.2-1, Clang 2.9-7, GCC 4.7 snapshot 20111105 (I built the LLVM 3.0 release aswell but it failed compiling stating it was unable to find libc++, I guess I'll wait for the full release to hit the Arch repos for that one)

Anyway there wasn't much to compare against this time in Phoronix's test since for some reason there was no Clang/LLVM test and Open64 (4.2.4-2) wasn't able to build p7zip on my machine. But as has always been the case in my tests, GCC outperforms Clang/LLVM by a good margin.

[XorEaxEax]

Oh and if anyone wants it, here's the (ugly, my bash-script-fu is very weak) bash script I threw together to do the test:

#!/bin/bash
name=p7zip_
version=9.20.1
dir=$name$version
results=../p7zip-results.txt
makefile=makefile.linux_amd64
arg="./bin/7za a -mx9 -m0=LZMA2:d64m -mmt dummy.7z archlinux-2011.08.19-netinstall-i686.iso"
flags="-O3 -march=native"
path64flags="-O3 -march=auto"

# uncomment compilers to be tested
gcc=1
clang=1
#gcc47=1
#path64=1


if [ ! -d $dir ]; then
	echo "Downloading and extracting p7zip..."
	wget -q http://downloads.sourceforge.net/project/p7zip/p7zip/${version}/p7zip_${version}_src_all.tar.bz2
	bzip2 -q -d "$dir"_src_all.tar.bz2
	tar xf "$dir"_src_all.tar
fi

cd $dir

if [ ! -e archlinux-2011.08.19-netinstall-i686.iso ]; then
	echo "Downloading iso..."
	wget -q http://ftp.ds.hj.se/pub/os/linux/archlinux/iso/2011.08.19/archlinux-2011.08.19-netinstall-i686.iso
fi


if [ ! -z $gcc ]; then
# GCC ----------------------------------------------------------------------------------------------------
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "GCC\t\t$flags\t\t$bench" >> $results
rm dummy.7z
# pgo
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-generate/" makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
rm dummy.7z
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-use -fprofile-correction/" makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "GCC\t\t$flags\t\t$bench\t\tProfile guided optimization" >> $results
rm CPP/7zip/Bundles/Alone/*.gcda
fi

if [ ! -z $gcc47 ]; then
# GCC 4.7 ----------------------------------------------------------------------------------------------------
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "GCC4.7\t\t$flags\t\t$bench" >> $results
rm dummy.7z
# pgo
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-generate/" makefile.machine
sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
rm dummy.7z
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags -fprofile-use -fprofile-correction/" makefile.machine
sed -i 's/CXX=g++/CXX=g++-4.7/' makefile.machine
sed -i 's/CC=gcc/CC=gcc-4.7/' makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "GCC4.7\t\t$flags\t\t$bench\t\tProfile guided optimization" >> $results
rm CPP/7zip/Bundles/Alone/*.gcda
fi

if [ ! -z $clang ]; then
# CLANG/LLVM ---------------------------------------------------------------------------------------------
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$flags/" makefile.machine
sed -i 's/CXX=g++/CXX=clang++/' makefile.machine
sed -i 's/CC=gcc/CC=clang/' makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "Clang\t\t$flags\t\t$bench" >> $results
rm dummy.7z
fi

if [ ! -z $path64 ]; then
# PATH64 ---------------------------------------------------------------------------------------------
cp $makefile makefile.machine
sed -i "s/OPTFLAGS=-O/OPTFLAGS=$path64flags/" makefile.machine
sed -i 's/CXX=g++/CXX=pathCC/' makefile.machine
sed -i 's/CC=gcc/CC=pathcc/' makefile.machine
make clean
make -j5
bench=`{ time ${arg}; } 2>&1 | grep real | awk '{print $2}'`
echo -e "Path64\t\t$path64flags\t\t$bench" >> $results
rm dummy.7z
fi

Put it in a directory and execute it, it will download p7zip and the iso used to compress the first time it's run and then perform the benchmarks and place the results in a textfile named p7zip-results.txt in the directory of the benchmark script. Just change the cflags in the script to those you want to test and also comment/uncomment the compilers you wish to test (note that open64 and path64 fails on p7zip on my machine). Also this assumes a 64bit system, if you want to benchmark on a 32-bit system you need to change the makefile variable to one of the 32-bit makefiles which accompany p7zip (weird buildsystem in p7zip, give me ./configure anyday!)