The Widely-Used Cairo Library Has Problems Passing Its Own Tests
Written by Michael Larabel in Free Software on 3 July 2016 at 09:59 AM EDT. 32 Comments
FREE SOFTWARE --
The Cairo graphics library that's relied upon by GTK+, Gecko/Firefox, WebKit, Poppler, and many other pieces of Linux software is having problems passing its own build tests.

Phoronix reader "Darxus" who previously wrote about reasons for losing motivation in Wayland, shared some details this week about Cairo's inability to pass its own build tests.

Darxus started a Cairo mailing list thread about the build test issues, to which it was acknowledged there are problems. He also wrote in an email to Phoronix explaining more details:
The Cairo test suite, by default, tests 16 "targets":

Target Tests Failures
image 1066 2
image16 533 1
ps 1
ps2 1066 7
ps3 1066 7
recording 1066 4
script 533 9
xcb 1066 6
xcb-fallback 533 1
xcb-render-0_0 1066 2
xcb-window 533 5
xcb-window& 533 5
xlib 1066 14
xlib-fallback 533 1
xlib-render-0_0 533 12
xlib-window 533 7

xcb and xlib are obviously the most important. "image" is something like simulated output using images. "ps" is postscript. "image" passes if you run it against Xvfb with 8 bits per pixel, but two tests crash if you run it at 24 bits. The only target that passes is the only one that has only a single test.

And this is if you regenerate all its own reference images. You're supposed to use the reference images that come with it, but I'm just trying to find a way to get it to pass, in desperation, to try to be able to catch any new failures. This really should make everything pass, since the references were generated exactly the same as the test output.

The most exciting part is that many of the failures are different each time - the diffs between the reference images and test output keep changing, when they shouldn't.

From the #cairo IRC channel:

[pq] Darxus, ok, just running code that is supposed to be deterministic twice and getting differing results is really bad. I think you can appreciate how bad it is, and unfortunately there is no quick solution.

[pq] Darxus, if you ran it with different Cairo backends, or different hardware or different drivers, that might explain some, but really identical runs - that's very bad.

[ebassi] It did use to pass its own test suite, until the downscaling algorithm was tweaked

[Darxus] ebassi: When was the downscaling algorith tweaked?

[ebassi] Darxus: Two or three years ago

[Darxus] ebassi: Did you see that the tests are passing / failing inconsistently? Any idea what it'll take to fix them?

[ebassi] No idea, sorry

Does it frighten you that the Cairo library that's so widely used throughout the Linux desktop ecosystem has issues going on for years relating to passing its own tests? Feel free to discuss in the forums.
About The Author
Author picture

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 10,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter or contacted via MichaelLarabel.com.

Related Free Software News
Popular News This Week