Don't use gstreamer-based sw then. It's widely known not to be fast.

Measure with mplayer or some other direct, ffmpeg-based app instead.