Frankly the problem is that input from a mouse, or a tablet or multiple touch-points is ambiguous. You may be painting in GIMP - not trying to "swipe to scroll". I can go on with examples (dragging a slider inside a scrollable area as opposed to wanting to scroll with a drag). Only the client has enough detailed knowledge of the window contents, application mode etc. To possibly make a reliable guess as to which one to do. It's X's job to provide as much device input to the client as it needs in a sane digestible way to make such a decision, but... that's [in my honest opinion] where the server's job ends."