Maybe. I've tried just pausing the video and waiting and the buffering didn't speed up. Firefox still has some performance issues that are currently being tweaked, but they've made AMAZING progress. They're already much better than flash on my computer. The downloading is the only issue that I experienced when I last tested it myself (and some youtube bug where full screen didn't do full screen but "full browser"). For all I know though, they could be fixed because I haven't tested it in a month or so.
Unless there is some new bug that causes that, that isn't the problem. If I wait, all of the videos would completely download.