Why TensorFlow Lite Has Been Running Slower On Recent Linux Kernels

[phoronix]

Phoronix: Why TensorFlow Lite Has Been Running Slower On Recent Linux Kernels

The Linux 5.0 to 5.9 kernel benchmarking posted this week showed TensorFlow Lite running slower since the Linux 5.5 kernel... On top of looking at the new Linux 5.9 regressions, I also spent some time bisecting and figuring out what happened for TensorFlow Lite last year that has at least for the system under test caused it to run slower for all the kernel releases this year as shown in the aforelinked article.

http://www.phoronix.com/vr.php?view=29486

[chuckula]

sched/fair: Remove meaningless imbalance calculation.

Looks like it was a little less meaningless than they thought!

[marlock]

Following the rule of "don't delete before understanding why it was put there in the first place" doesn't seem hard with such a small and seemingly well-commented code block.

To be fair:
- i don't know what I'm talking about, not a kernel proggramer
- I do understand that any one-liner change on the linux kernel can potentially have horribly complex ramifications
- I did read recently on how scheduler code can spider over a lot of places in the kernel in an attempt to gather enough data to execute adequate load balancing, etc

But, from my layman view, this comment on one of the deleted pieces of the code does look like it was put there for a good reason... to improve performance on specific loads, so it *looks* a bit unsurprising that taking it out regressed performance on specific loads:


"[...] OK, we don't have enough imbalance to justify moving tasks, [...] however we may be able to increase total CPU capacity used by [...] moving them."

Anyone with more in-depth knowledge care to comment on the whys and whynots?

[DanglingPointer]

Good forensic work Michael!

With the scheduler being such a crucial part of the kernel, you would think they would run perf tests on changes before approving it to mainline!

[Space Heater]

With the scheduler being such a crucial part of the kernel, you would think they would run perf tests on changes before approving it to mainline!

They did and it seems that the set of changes improves performance in some cases, and for this workload it caused a performance regression.

[Michael]

With the scheduler being such a crucial part of the kernel, you would think they would run perf tests on changes before approving it to mainline!

Part of the issue is the shear number of different workloads to test. (Though then again with PTS are 500+ different benchmarks at easy disposal.)

[marlock]

I have to say, Michael!

You are an instituition, and your work on this front may cost more in power bills than it pays in ads for now, but maybe it's what will finally push PTS into evidence for continuos integration suites and get you a well earned monthly paycheck.

That and you may well be the one person with the most diverse set of hardware in a single place near someone-who-knows-what-to-do (tm) for these new linux hardware testing innitiatives.

How well does PTS integrate into current CI solutions by the way?

[Michael]

How well does PTS integrate into current CI solutions by the way?

It depends what CI solutions you are referring to but I have heard of people plugging it into Jenkins and others.

PTS' Phoromatic can already facilitate testing on a daily/timed basis or other factors like per-commit basis via setting up a simple ping to a given Phoromatic URL. So with being able to trigger new tests to run based on a customized URL, it's generally very easy to integrate PTS/Phoromatic into custom environments.

Anything more could be easily achieved via custom engineering work happy to pursue, but given the lack of that in this area, I am imagining the existing integration support is working out well enough at the different organizations.

[Brutalix]

Bisecting regressions is what triggers additional donations from me. Keep up the good work.
B

[rene]

this should have been obviously from the beginning, and certainly be long plumbed already ¯\_(ツ)_/¯.