From my POV the two problems are very much related. In fact so much that I would handle it all through one ioctl.
In both cases you need:
- pass an arbitrary number of properties to the driver to modify the state
- the full device state must available to the driver before you can check whether a certain configuration is possible
- the API must support a mode where you just check the state, but don't apply it to the hardware
Where things start to differ is when you want to modify the actual hardware state.
In the pagelip case you absolutely want two features in the apply path:
- asynchronouse operation to avoid blocking the compositor
- atomically apply the state within a single hardware pipe/crtc
Ideally you would also want asynchronous execution for the modeset path, but people are lazy and don't want to write the necessary state machine (or whatever logic) to handle it. So that's why mode setting is generally is handled through a synchronous (ie. blocking) code path.
Now, since you anyway need the same state checking and virtually identical API in both cases, I see no reason to impose a split on that level. It can all be left as an internal implementation detail. Doing it that way would also allow some enterprising developer to actually implement asynchronous mode setting if they so wish.
So as I see it, you just need a few extra flags in the API so that user space can demand asynchronous operation from the driver. If the driver can't satisfy the request asynchronously the user gets an error, and can do whatever fallback is appropriate.