The Widely-Used Cairo Library Has Problems Passing Its Own Tests

The Cairo graphics library that's relied upon by GTK+, Gecko/Firefox, WebKit, Poppler, and many other pieces of Linux software is having problems passing its own build tests.

Phoronix reader "Darxus" who previously wrote about reasons for losing motivation in Wayland, shared some details this week about Cairo's inability to pass its own build tests.

Darxus started a Cairo mailing list thread about the build test issues, to which it was acknowledged there are problems. He also wrote in an email to Phoronix explaining more details:

The Cairo test suite, by default, tests 16 "targets":

Target Tests Failures
image 1066 2
image16 533 1
ps 1
ps2 1066 7
ps3 1066 7
recording 1066 4
script 533 9
xcb 1066 6
xcb-fallback 533 1
xcb-render-0_0 1066 2
xcb-window 533 5
xcb-window& 533 5
xlib 1066 14
xlib-fallback 533 1
xlib-render-0_0 533 12
xlib-window 533 7

xcb and xlib are obviously the most important. "image" is something like simulated output using images. "ps" is postscript. "image" passes if you run it against Xvfb with 8 bits per pixel, but two tests crash if you run it at 24 bits. The only target that passes is the only one that has only a single test.

And this is if you regenerate all its own reference images. You're supposed to use the reference images that come with it, but I'm just trying to find a way to get it to pass, in desperation, to try to be able to catch any new failures. This really should make everything pass, since the references were generated exactly the same as the test output.

The most exciting part is that many of the failures are different each time - the diffs between the reference images and test output keep changing, when they shouldn't.

From the #cairo IRC channel:

[pq] Darxus, ok, just running code that is supposed to be deterministic twice and getting differing results is really bad. I think you can appreciate how bad it is, and unfortunately there is no quick solution.

[pq] Darxus, if you ran it with different Cairo backends, or different hardware or different drivers, that might explain some, but really identical runs - that's very bad.

[ebassi] It did use to pass its own test suite, until the downscaling algorithm was tweaked

[Darxus] ebassi: When was the downscaling algorith tweaked?

[ebassi] Darxus: Two or three years ago

[Darxus] ebassi: Did you see that the tests are passing / failing inconsistently? Any idea what it'll take to fix them?

[ebassi] No idea, sorry

Does it frighten you that the Cairo library that's so widely used throughout the Linux desktop ecosystem has issues going on for years relating to passing its own tests? Feel free to discuss in the forums.