I think your making the grand assumption that my 3 (or even 4) cores aren't already busy with other processing. It is quite typical on my desktop to be running 30-40 applications (all proaudio, using/requiring RT-scheduling), right now, *idling* i am using 34% of my CPU(s), keep in mind that i am running 28 proaudio applications, not including Firefox, gedit and musescore. Not one of my cores is idling at zero (or even close to zero). As for the latter part of your comment, not sure, wouldn't want to speculate either.
Wow! That's more overhead than even I thought rt would have, based on our earlier discussions.
Have you tried adjusting the slack timer so it can avoid some wakeups?