compilers need some linguistics ("how do I parse this stuff?"), a lot of theory of computation (i.e. "prove that these two code snippets produce identical output" and other problems), lots of data structures ("how do I organize the information about the code so it's quick to access and modify, but still easy to handle?", basically the question of picking good IRs), a lot of technics specific to compilers (that have to be invented, implemented and tested) and a whole bunch of other skills. Basic math is included in the latter, but nowhere near a dominant skill.
What I would find interesting is what CFLAGS are used in the compilation. And as he said ha ran the "default" I do not think he went around adding flags in different places and hacked the buildsystem to include "-fwhole-program and alike. I assume that the sources are built somewhat like "unset CFLAGS && ./configure && make", and in that case, what compiler flags are the buildsystem adding and what differs between them is more interesting.
The biggest differences between versions in gcc when it comes to general optimization seems to be that different -O# flags have different options integrated, so if you do not specify a flag, and the buildsystem for one program does not add any -O# that program may not show as much "difference" between gcc releases as one adding -O2.
On x86 targets, code containing floating-point calculations may run significantly slower when compiled with GCC 4.5 in strict C99 conformance mode than they did with earlier GCC versions. This is due to stricter standard conformance of the compiler and can be avoided by using the option -fexcess-precision=fast; also see below.
When you run your performance tests, do you talk about your results with project developers?
... Basic math is included in the latter, but nowhere near a dominant skill.
To make a working compiler you do not need much math.
To make a splendid compiler you really need to know your math.
This is because much optimization in all code really is to know how to in as few steps as possible do as much as possible. And sometimes that includes knowing things like mathematically what advantages and disadvantages different sort-algorithms has and on what load.
But OTOH I do sometimes count knowing stuff like derivative as "basic" math-skills.