In-Kernel Power Management For ATI KMS

[phoronix]

Phoronix: In-Kernel Power Management For ATI KMS

While the Radeon R100-R500 series kernel mode-setting support appeared in the Linux 2.6.31 kernel and DRM patches pending for the Linux 2.6.32 kernel that bring KMS support for newer hardware and other improvements, the ATI KMS driver is not complete. Features such as power management need to be brought into the kernel driver (for Intel too) where they will be better off compared to the traditional DDX drivers...

http://www.phoronix.com/vr.php?view=NzUyOA

[Louise]

I sure hope these patches can make it into Fedora 12 :)

[kernelOfTruth]

simply incredible :eek:

Rafał Miłecki and all the other devs:

thank you, thank you, thank you, thank you, thank you very much :)

I can't believe how fast everything is coming together !

great work guys ! :cool:

[DoDoENT]

I sure hope these patches can make it into Fedora 12 :)

I hope that too. My fedora laptop will finally have longer battery life. :D

[Zajec]

Phoronix idealized a little my patches :) I think it sounds too optimistic ;)

My work prepares only basic infrastructure for PM. This doesn't do anything useful yet actually.

Last patch which introduces sth interesting - downclocking - is considered not ready yet. I think it may even crash module, as it doesn't even check function pointer.

As soon as these patches hit drm-next, I'm going to continue my work and prepare something really useful. Unfortunately the most complicated part is still ahead of us: determining which power state should be used in current moment. It needs measuring current load and some smart (and lucky) choosing mode. Remember that every mode has engine clock, memory clock and voltage. We have to somehow calculate all of that.

So I don't think you should expect something "really" working soon. Especially if you keep in mind that main (much more talented and experienced) developers are still working on other parts.

[bugmenot]

Thanks, Zajec! :)

Do you know by chance something about reading out temperatures? I am very interested in that.

[whizse]

As soon as these patches hit drm-next, I'm going to continue my work and prepare something really useful. Unfortunately the most complicated part is still ahead of us: determining which power state should be used in current moment. It needs measuring current load and some smart (and lucky) choosing mode. Remember that every mode has engine clock, memory clock and voltage. We have to somehow calculate all of that.

Isn't this something which could be lifted from the proprietary driver?

If not the actual code, then the calculations behind it? I would guess having the cards behave the same between the two drivers would be a good idea?

[Ant P.]

Unfortunately the most complicated part is still ahead of us: determining which power state should be used in current moment.

Will it still have the behaviour of the current radeon driver where it downclocks during DPMS power off? That one makes a huge difference even on my 4350.

Other than that, I imagine something like cpufreq ondemand would work, if the driver had a way to detect the framerate of running apps and change the speed as appropriate.

[lucky_]

Phoronix idealized a little my patches :) I think it sounds too optimistic ;)

My work prepares only basic infrastructure for PM. This doesn't do anything useful yet actually.

Last patch which introduces sth interesting - downclocking - is considered not ready yet. I think it may even crash module, as it doesn't even check function pointer.

As soon as these patches hit drm-next, I'm going to continue my work and prepare something really useful. Unfortunately the most complicated part is still ahead of us: determining which power state should be used in current moment. It needs measuring current load and some smart (and lucky) choosing mode. Remember that every mode has engine clock, memory clock and voltage. We have to somehow calculate all of that.

So I don't think you should expect something "really" working soon. Especially if you keep in mind that main (much more talented and experienced) developers are still working on other parts.
Well first thanks for the work, even though it's in an early stage.
But I have a couple of things in my mind, here they are :
As I understood it from the previous discussions here on phoronix, the really tricky part was to ensure that the down-clocking was safe.
By safe I mean that such down-clocking can raise issue with the memory being corrupted and thus some flushing was needed.

At the beginning you can always build rather simple heuristic to test how your set-up works. (for instance a function that follows the inverse of the temperature curves, really stupid since every state change should induce a change in the temperature as well, which in turn should trigger a new state change, etc...)
Tuning such algorithms takes time and you only end up knowing exactly what to do (and what people want you to do) once people start to complain that your implementation is too aggressive or too lazy.

By looking at the radeon_pm_set_state function, I see that you skipped the memory and voltage setting, isn't it a problem ? Are the frequencies of the components paired ? Or can you access the memory at every frequency from an engine at any given frequency ?
I might try to implement some fancy idea in your radeon_pm_adjust, if it's proven to safe from the point of view of the kernel.

[Zajec]

Isn't this something which could be lifted from the proprietary driver?

If not the actual code, then the calculations behind it? I would guess having the cards behave the same between the two drivers would be a good idea?I'm afraid AMD is too busy with other things. For example Christian is waiting for HDMI audio specs (even if under NDA) for about month already.



Will it still have the behaviour of the current radeon driver where it downclocks during DPMS power off? That one makes a huge difference even on my 4350.That will be the easiest part and should be implemented as first PM operation. Detecting DPMS_OFF -> downclocking to minimum. Nothing could be easier :) Actually you can expect this one pretty soon (still, depends on amount on changes Alex will want to make to my patches).



Other than that, I imagine something like cpufreq ondemand would work, if the driver had a way to detect the framerate of running apps and change the speed as appropriate.If driver can detect FPS, it's something I don't know about. And probably even if, we could not use this as the only parameter to choose state. I think we will need to try observe commands submissions.

There already were some ideas posted on ML/IRC, will have to grab them all and consider.



lucky_: will read you post tomorrow, sorry :) Goodnight.

[lucky_]

No worries goodnight.

[bridgman]

By looking at the radeon_pm_set_state function, I see that you skipped the memory and voltage setting, isn't it a problem ? Are the frequencies of the components paired ? Or can you access the memory at every frequency from an engine at any given frequency ?

Memory clocks are more complicated than engine clocks, in the sense that memory timing depends on both engine activity and display activity (number of screens, resolution/refresh rate of each display, image depth, image tiling etc..).

Prior to having KMS it wasn't really practical to play with memory speeds since there was no place in the driver stack which had access to 2D, 3D, video and display activity information (unless a new protocol were defined to pass display info down to the kernel).

You also need to block all display and drawing activity when changing memory clocks to avoid corruption and again that could only be done in a KMS DRM.

Adjusting memory clocks on a GDDR5 system is more complicated because the memory subsystem needs to go through a training cycle when clocks are changed, and that can sometimes be too long to fit into a display blanking period (along with the obvious problem of what to do with two displays running at different refresh rates).

[Louise]

Adjusting memory clocks on a GDDR5 system is more complicated because the memory subsystem needs to go through a training cycle when clocks are changed, and that can sometimes be too long to fit into a display blanking period (along with the obvious problem of what to do with two displays running at different refresh rates).

I hope this isn't a too stupid question. But why is power management done in software on the cpu?

Couldn't the GPU "power manage" itself in hardware?

[bridgman]

Actually a very good question.

In principle, yes the hardware could be completely self sufficient for power management.

In practice, however, making the right power management decisions usually requires knowledge outside what the GPU or CPU has available. As an off-the-top-of-my-head example, you want to power down the 3D engine very aggressively when there is no drawing activity, but if your game stalls while swapping in a big texture you probably would *not* want to power down the 3D engine in that case. That decision heuristic would require that power management for the GPU be cognizant of disk activity.

The preferred approach for managing power is also changing over time, with a slight trend from "run the hardware as slowly as possible while maintaining acceptable performance" to "run at full speed, get the work done fast and then switch everything off as quickly as possible until more work comes along". As you can imagine, it helps a lot of the CPU and GPU power management follow the same approach :D

When setting memory clocks you need to make some pretty complex bandwidth and latency calculations in order to make sure that the display fifos don't get starved for data when the 3D engine is drawing into the same memory. One implication of this is that the minimum memory clock setting is a function of engine clock, drawing activity, number of active displays, depth / resolution / refresh rate.

Since changing clocks takes a non-trivial amount of time (you need to let the PLLs in the clock generators settle at the new frequency after changing dividers) you need to include recent behaviour trends in the decision-making logic, not just what is happening at this instant, in order to avoid too-frequent changes with the associated waste of power and time. All of this can be done in hardware but you can see how quickly the complexity can grow.

Anyways, bottom line is getting the very best power efficiency always seems to require making decisions using more information than what is available to each individual device, ie shifting from focusing on device power management to focusing on platform power management.

[Zajec]

Thanks, Zajec! :)

Do you know by chance something about reading out temperatures? I am very interested in that.Don't know. I just checked AtomBIOS commands but don't see anything related to temperature there. Just
GPIOPinControl command makes me wondering. Can sb explain what for is GPIO used on ATI cards?
So we probably have to use some magic not-yet-known register to get temperature.



lucky_: OK, I see bridgman replied to you.


I still don't know how we should calculate load. Someone posted yesterday idea of checking command buffer (it's full → GPU too slow). Don't know much about that, have to check how CS works.

Also I don't know what modes we can use for whole clocking. Anything between minimum and maximum? Or only states found in AtomBIOS?

[DoDoENT]

As soon as these patches hit drm-next, I'm going to continue my work and prepare something really useful. Unfortunately the most complicated part is still ahead of us: determining which power state should be used in current moment. It needs measuring current load and some smart (and lucky) choosing mode. Remember that every mode has engine clock, memory clock and voltage. We have to somehow calculate all of that.


I'm not the expert, but tell my why is important for driver to choose the best power state to be used in current moment? Isn't just possible to make some file, such as /sys/*/radeon/power_state which will control the current power state? I mean: if I write into terminal:
cat /sys/*/radeon/power_state
it should print something like the following:
Current power state: 2 (x MHz Core Clock, y MHz Memory Clock)
And I could change the power state by entering e.g.: echo 1 > /sys/*/radeon/power_state
And available power states for current graphics card would be available in another file, e.g. /sys/*/radeon/power_states, which would be readonly.

In that case, determining which power state should be used in current moment could be left to the higher-level programs, e.g. PowerDevil, laptop-mode, or some custom user scripts.

Is it possible to enable this way of switching power states in radeon driver? In this way works the iwl3945 drivers's power management (of course iwl3945 is the WiFi driver, and I'm just curious if it's possible to create the similar power management architecture for graphics driver as is for wifi driver?)

Thank you in advance for your answers.

[Zajec]

I'm not the expert, but tell my why is important for driver to choose the best power state to be used in current moment? Isn't just possible to make some file, such as /sys/*/radeon/power_state which will control the current power state? I mean: if I write into terminal:
cat /sys/*/radeon/power_state
it should print something like the following:
Current power state: 2 (x MHz Core Clock, y MHz Memory Clock)
And I could change the power state by entering e.g.: echo 1 > /sys/*/radeon/power_state
And available power states for current graphics card would be available in another file, e.g. /sys/*/radeon/power_states, which would be readonly.

In that case, determining which power state should be used in current moment could be left to the higher-level programs, e.g. PowerDevil, laptop-mode, or some custom user scripts.

Is it possible to enable this way of switching power states in radeon driver? In this way works the iwl3945 drivers's power management (of course iwl3945 is the WiFi driver, and I'm just curious if it's possible to create the similar power management architecture for graphics driver as is for wifi driver?)

Thank you in advance for your answers.What you talk about is userspace power management. This can be eventually introduced, but can not be the only solution.

Userspace just can not know about everything that happens in GPU. It has only some basic idea about current load (amount of apps, maybe Xv and 3D detecting). That's not enought. Only kernel side can have full view on load.

However as I said, we will want to introduce something you proposed anyway. For example in case of using battery power we don't know if user still prefers performance, or long battery life.

For some additional details you may also check discussion on dri-devel: http://sourceforge.net/mailarchive/message.php?msg_name=b170af450909090150s6da5803bj7 5404cdd9f5705ba%40mail.gmail.com

[bridgman]

What Zajec said ;)

Doing simple power management in userspace is no problem, but doing the kind of aggressive real-time power management that you need for best battery life on a laptop either requires the decision making code to be in the kernel or a protocol to be defined that passes all the information up to userspace so that information can be accumulated and decisions made there.

It doesn't really matter where the other than (a) it's easier for userspace code to have access to user inputs etc,, (b) all the information comes from the kernel and decisions end up in the kernel so doing the first implementation *in* the kernel is probably less work unless all of the necessary user/kernel protocols already exist (which they might, of course).

[Louise]

Actually a very good question.

In principle, yes the hardware could be completely self sufficient for power management.

In practice, however, making the right power management decisions usually requires knowledge outside what the GPU or CPU has available. As an off-the-top-of-my-head example, you want to power down the 3D engine very aggressively when there is no drawing activity, but if your game stalls while swapping in a big texture you probably would *not* want to power down the 3D engine in that case. That decision heuristic would require that power management for the GPU be cognizant of disk activity.

The preferred approach for managing power is also changing over time, with a slight trend from "run the hardware as slowly as possible while maintaining acceptable performance" to "run at full speed, get the work done fast and then switch everything off as quickly as possible until more work comes along". As you can imagine, it helps a lot of the CPU and GPU power management follow the same approach :D

When setting memory clocks you need to make some pretty complex bandwidth and latency calculations in order to make sure that the display fifos don't get starved for data when the 3D engine is drawing into the same memory. One implication of this is that the minimum memory clock setting is a function of engine clock, drawing activity, number of active displays, depth / resolution / refresh rate.

Since changing clocks takes a non-trivial amount of time (you need to let the PLLs in the clock generators settle at the new frequency after changing dividers) you need to include recent behaviour trends in the decision-making logic, not just what is happening at this instant, in order to avoid too-frequent changes with the associated waste of power and time. All of this can be done in hardware but you can see how quickly the complexity can grow.

Anyways, bottom line is getting the very best power efficiency always seems to require making decisions using more information than what is available to each individual device, ie shifting from focusing on device power management to focusing on platform power management.

Okay that's just crazy the things that is going one behind the scene :)

I assume what you just described is the most advanced and probably the latest to get implemented? ;)

So KMS is power management's hook into the 3D engine/OpenGL*?

I assume 3D engine and OpenGL is the same...?

[DoDoENT]

What Zajec said ;)

Doing simple power management in userspace is no problem, but doing the kind of aggressive real-time power management that you need for best battery life on a laptop either requires the decision making code to be in the kernel or a protocol to be defined that passes all the information up to userspace so that information can be accumulated and decisions made there.

It doesn't really matter where the other than (a) it's easier for userspace code to have access to user inputs etc,, (b) all the information comes from the kernel and decisions end up in the kernel so doing the first implementation *in* the kernel is probably less work unless all of the necessary user/kernel protocols already exist (which they might, of course).

I understand.

But how about enabling just userspace power state switching by the time Fedora 12 ships? It would mean a lot to advanced users which would like to have at least some power management (better that than none). I know this wouldn't offer the best battery life for laptops, but it will increase it... at least as it was while I was using fglrx (the difference is approx. 35-40 minutes).

[bridgman]

Louise;

the KMS drm driver is important in two ways :

1. As you say, it provides the "hook" into all 3D engine use, both from the OpenGL driver (the obvious user of the 3D engine) and the X driver (which uses the 3D engine for EXA and Xv as well). It also provides access to display / modesetting info in the same driver, ie it brings all of the required information together in one place.

2. Some of the registers which control GPIO and I2C lines for reading and writing fan/temp controllers and voltage control are also used by modesetting for reading EDID information, so modesetting and power management need to be in the same driver. Unfortunately that needs to be the drm driver, since changing clocks on the fly requires that the driver also block any use of drawing engines, which can only be done in the drm driver (since direct rendered 3D doesn't go through the userspace X driver).

DoDoENT;

The problem with doing dynamic PM in the userspace driver is that the userspace driver can't block drawing calls from 3D. Bad things can happen if the drawing engine is running at the same time you are reprogramming the clock generator for the engine. Doing dynamic PM in the drm means that drawing operations can be temporarily stopped and the drawing engine quiesced before changing the clock.

[Louise]

Louise;

the KMS drm driver is important in two ways :

1. As you say, it provides the "hook" into all 3D engine use, both from the OpenGL driver (the obvious user of the 3D engine) and the X driver (which uses the 3D engine for EXA and Xv as well). It also provides access to display / modesetting info in the same driver, ie it brings all of the required information together in one place.

2. Some of the registers which control GPIO and I2C lines for reading and writing fan/temp controllers and voltage control are also used by modesetting for reading EDID information, so modesetting and power management need to be in the same driver. Unfortunately that needs to be the drm driver, since changing clocks on the fly requires that the driver also block any use of drawing engines, which can only be done in the drm driver (since direct rendered 3D doesn't go through the userspace X driver).


Very interesting all the things that KMS can be used for!

[DoDoENT]

DoDoENT;

The problem with doing dynamic PM in the userspace driver is that the userspace driver can't block drawing calls from 3D. Bad things can happen if the drawing engine is running at the same time you are reprogramming the clock generator for the engine. Doing dynamic PM in the drm means that drawing operations can be temporarily stopped and the drawing engine quiesced before changing the clock.

I see. So the AI in the driver is required after all, and it should make decision based on the user preferences from the userspace (i.e. does the user want max performance or max powersave).

What I have in mind is not dynamic PM from userspace, but static instead. The user would set a request that from now on, he wants maximum performance, as he (or she) is going to play a game. Then, the PM AI would make decisions which would offer best performance, but not the best power saving. And if the user sets request for maximum power saving, the PM AI should make decisions which would offer the least performance, but best power saving. And finally, the user would have the option to set a request for smart power saving, which would then do what you've told - it would make smart decisions which will optimize the ratio between performance and power saving based on some data it collects. This third part would be the most difficult to make as it requires relatively complex AI algorithms, but even first two parts will make a lot of people happy (including me :D).

Just to make myself clear: I would like to have the famous aticonfig --set-powerstate feature in the radeon driver: when I issued "aticonfig --set-powerstate=1", I got 3 hours of battery life with awful graphics performance (but good enough to do some simple jobs), and when I issued "aticonfig --set-powerstate=3", I got less than 1 hour of battery life, but I could play nexuiz with high details in high resolution without any problems. This is what I find very useful, and it doesn't look like it requires any complex AI PM algorithms.

[Zajec]

What do you think about creating something like /sys/class/gpu/ with engine_clock, memory_clock and voltage? Example:$ cat /sys/class/gpu/engine_clock
management: auto
300000 KHz

$ echo maximum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
680000 KHz

$ echo minimum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 50000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 250000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
250000 KHz

$ echo auto > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: auto
320000 KHz?

[suokko]

What do you think about creating something like /sys/class/gpu/ with engine_clock, memory_clock and voltage? Example:$ cat /sys/class/gpu/engine_clock
management: auto
300000 KHz

$ echo maximum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
680000 KHz

$ echo minimum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 50000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 250000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
250000 KHz

$ echo auto > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: auto
320000 KHz?

We would still need UI for that like gnome-applet or plasma widget.

[suokko]

I see. So the AI in the driver is required after all, and it should make decision based on the user preferences from the userspace (i.e. does the user want max performance or max powersave).

What I have in mind is not dynamic PM from userspace, but static instead. The user would set a request that from now on, he wants maximum performance, as he (or she) is going to play a game. Then, the PM AI would make decisions which would offer best performance, but not the best power saving. And if the user sets request for maximum power saving, the PM AI should make decisions which would offer the least performance, but best power saving. And finally, the user would have the option to set a request for smart power saving, which would then do what you've told - it would make smart decisions which will optimize the ratio between performance and power saving based on some data it collects. This third part would be the most difficult to make as it requires relatively complex AI algorithms, but even first two parts will make a lot of people happy (including me :D).

Just to make myself clear: I would like to have the famous aticonfig --set-powerstate feature in the radeon driver: when I issued "aticonfig --set-powerstate=1", I got 3 hours of battery life with awful graphics performance (but good enough to do some simple jobs), and when I issued "aticonfig --set-powerstate=3", I got less than 1 hour of battery life, but I could play nexuiz with high details in high resolution without any problems. This is what I find very useful, and it doesn't look like it requires any complex AI PM algorithms.

I think we don't need any more complex algorithm for smart than cpufreq has. It is just a bit harder to calculate load level for GPU.

[Zajec]

We would still need UI for that like gnome-applet or plasma widget.Sure, that should eventually be last step.

[DoDoENT]

What do you think about creating something like /sys/class/gpu/ with engine_clock, memory_clock and voltage? Example:$ cat /sys/class/gpu/engine_clock
management: auto
300000 KHz

$ echo maximum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
680000 KHz

$ echo minimum > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 50000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
110000 KHz

$ echo 250000 > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: static
250000 KHz

$ echo auto > /sys/class/gpu/engine_clock
$ cat /sys/class/gpu/engine_clock
management: auto
320000 KHz?

Exactly my idea!

[DoDoENT]

We would still need UI for that like gnome-applet or plasma widget.

That's not difficult. I mean: That's _really_ not difficult. :D

[DoDoENT]

I think we don't need any more complex algorithm for smart than cpufreq has. It is just a bit harder to calculate load level for GPU.

So, which information does the driver have for calculating the GPU load?

[agd5f]

So, which information does the driver have for calculating the GPU load?

Look at the incoming command stream and engine idle status.

[duby229]

So, which information does the driver have for calculating the GPU load?

That is actually part of the debate I think. As of right now the best ideas I've heard include monitoring the GPU's Command Processor to determine how much load is being put on the chip.

Here is a block diagram of r600 architecture.... Just scroll about a quarter of the page down

http://ixbtlabs.com/articles2/video/r600-part1.html

Now I'm not entirely sure, but from looking at that diagram it seems to me that monitoring the command processor would give you a good idea of general load on the gpu, but I dont think it will be able to tell what type of load it is or what its doing.

I guess my question is simple..... Is it possible to profile GPU loads?

[agd5f]

You can check the busy status of the various blocks, however the best way to determine the "load" is to track the amount of queued commands. The engine is either running or not.

[suokko]

That is actually part of the debate I think. As of right now the best ideas I've heard include monitoring the GPU's Command Processor to determine how much load is being put on the chip.

Here is a block diagram of r600 architecture.... Just scroll about a quarter of the page down

http://ixbtlabs.com/articles2/video/r600-part1.html

Now I'm not entirely sure, but from looking at that diagram it seems to me that monitoring the command processor would give you a good idea of general load on the gpu, but I dont think it will be able to tell what type of load it is or what its doing.

I guess my question is simple..... Is it possible to profile GPU loads?


There is no public documention for performance counter registers. I know at least nvidia blobs offering very good GPU profiling tools for developers. (I have seen some PR video showing GPU profiling in game)

I guess AMD has similar registers to read the performance statistics. But that would require drier support like a lot else so it is long way before even documentation would be used to provide interface for application developers.