VAAPI Segmentation fault right after va_getDriverName()

[markg85]

Hi,

It's that time again that i try out VAAPI with XvBA and yet again it manages to take nearly a full day of screwing around trying every possible combination to get the vainfo command running. Here is the output i get:

libva: libva version 0.31.1
Xlib: extension "XFree86-DRI" missing on display ":0.0".
libva: va_getDriverName() returns 0
Segmentation fault

I tried multiple libva versions. The one provided in the archlinux repo, the git version and the one provided on splitted desktop systems. None seem to work.

I also have xvba-video 0.7.8 installed and the latest catalyst (11.2) is installed. I'm completely out of ideas to try and get this darn thing running.

Anyone any ideas that i can try? I'm running archlinux fully updated.

Regards,
Mark

ps. Why isn't this seg fault fixed..? I mean, it's not like it's a "new" issue since if you search for it on google it's seems to happen a LOT!

[markg85]

AWESOME! I found a total of 3!!! freaking bugs!

1. if libva passes a null pointer to va_openDriver for the "driver_name" then the malloc function below will segfault! However, that is the effect. The cause of the effect is that the function is entered when it should not do that. For that i made a patch.

VAStatus vaInitialize (
    VADisplay dpy,
    int *major_version,     /* out */
    int *minor_version      /* out */
)
{
    const char *driver_name_env = NULL;
    char *driver_name = NULL;
    VAStatus vaStatus;

    CHECK_DISPLAY(dpy);

    va_TraceInit(dpy);

    va_FoolInit(dpy);

    va_infoMessage("libva version %s\n", VA_VERSION_S);

    driver_name_env = getenv("LIBVA_DRIVER_NAME");
    if (driver_name_env && geteuid() == getuid())
    {
        /* Don't allow setuid apps to use LIBVA_DRIVER_NAME */
        driver_name = strdup(driver_name_env);
        vaStatus = VA_STATUS_SUCCESS;
        va_infoMessage("User requested driver '%s'\n", driver_name);
    }
    else
    {
        vaStatus = va_getDriverName(dpy, &driver_name);
        va_infoMessage("va_getDriverName() returns %d\n", vaStatus);
    }

    if (VA_STATUS_SUCCESS == vaStatus && driver_name != 0)
    {
        vaStatus = va_openDriver(dpy, driver_name);
        va_infoMessage("va_openDriver() returns %d\n", vaStatus);

        if (vaStatus != VA_STATUS_SUCCESS && !driver_name_env)
        {
            if (va_hasCrystalHD())
            {
                vaStatus = va_openDriver(dpy, "crystalhd");
                va_infoMessage("va_openDriver() returns %d\n", vaStatus);
            }
        }

        *major_version = VA_MAJOR_VERSION;
        *minor_version = VA_MINOR_VERSION;
    }
    else
    {
        vaStatus = VA_STATUS_ERROR_UNKNOWN;
    }

    if (driver_name)
        free(driver_name);
    return vaStatus;
} 

Really, shame on the creators for leaving this in so long!
Oh and btw. I don't know the inner workings of libva but the function above (vaInitialize) really doesn't need to fetch the driver_name_env to driver_name.. Just fetch it, chec if it's not null and proceed if it has some value. My "patch" in that regards is not fine since i don't fix that but i'm not the libva dev so it's up to those people to fix this. Note: my patch is just a quick fix to prevent it from segfaulting!

That was bug 1 and the most severe bug.
Bug 2 was a unset LIBVA_DRIVERS_PATH
Bug 3 was a unset LIBVA_DRIVER_NAME

To be complete. Just setting those 2 defines like so (in my case):
export LIBVA_DRIVER_NAME=xvba
export LIBVA_DRIVERS_PATH=/usr/lib/dri/

solved my issue and vainfo works again.