Ok, so you're talking about learning to tune Linux for whatever workload. From the way you describe it, you should be unhappy with Solaris, since it can switch process schedulers at run time, so you have to learn how both of them behave, and interact with any other tunables and selectable versions of algorithms, right?Regarding if only a single (Solaris) binary to be considered as scalable or if modifying the (Linux) code to suit different machines to be considered as scalable. If you need to recompile and modify, then to me it is not scalable - because - I want to learn only one technology and one behaviour. If the different Linux kernels differ much, then I have to learn about them each. "Oh, now I am at cluster, then I must remember these things.... Oh, now I am at a desktop PC, then I must remember these things.... etc". Not only do you have to learn the different kernel versions, but within the same kernel version the kernels can behave differently and have different weak points and strong points. This messes my poor head. In Solaris, I only learn one kernel. And it works for whatever I want to do. I want to learn one technique very well and become a guru on it. It is easier for my poor head and memory. This is the reason, it is important to me personally to have only one kernel with only one behaviour no matter what you do.
None of Linux's compile-time config options will require learning new behaviour, since they don't change behaviour (except SLUB vs. SLAB). The harder thing to learn is all the run-time tunable variables and switches. (e.g. vm/swappiness, which many people disagree on good settings for...) If you like, think of the config options as just more tunables, which happen to be compile-time not run-time. Back in the day, a lot more tunables were compile time. That made sense when CPU instruction throughput was way more of a bottleneck compared to memory, and when caches were smaller. (i.e. before P6)
I just recently customized a kernel for my desktop, so I went and looked over the menu options in make xconfig (for amd64) and there aren't any alternate algorithms you can choose for big iron. What you'd actually do:
0. do some tuning on the stock kernel from your distro, unless it is totally unsuitable for your machine and what you learn from it would be useless. (e.g. Ubuntu's amd64 configs don't enable this, surprisingly. I guess they don't expect to get any use on dual-socket K8? weird. I'm going to file a bug report if it hasn't already come up. -server should have NUMA support.)
1. choose the actual CPU type (core2, Opteron, or P4 instead of generic x86-64). I think this mostly affects what gcc options are used.
2. enable NUMA memory allocation/scheduler awareness. Enable support for anything your machine has that isn't enabled by default.
3. Turn off all accounting, debugging, and statistics that you don't need/want for tuning or to leave available in production.
4. disable device drivers you don't need. This isn't a big deal, since the default config has almost everything built as modules. Memory consumption from dead code (core framework support for e.g. webcams) doesn't matter. It won't get any action, so it won't pollute your I-cache, which is what does matter. Maybe disabling some parts of kernel infrastructure will improve code locality by taking out dead code between code that will see some use.
5. compile and boot the kernel and start tweaking /proc/sys tunables.
6. Turn off all the stats gather stuff you don't need anymore, and build a production kernel. Although it's probably pretty easy to not do your stats gathering inside critical sections, so compiling it in shouldn't worsen scalability, just provide a tiny non-#CPU-dependent slowdown. And a bigger slowdown when stats gather is actually enabled with it's /proc/sys controls. (Most expensive stats gathering has to be enabled at runtime as well as compiled in.)
BTW, I lied. There is one compile-time algorithm selection: SLUB or SLAB kernel memory allocator. SLAB is the old one, and they keep it around because it's "known not to work well in all environments". So if your workload hits a corner case in SLUB, you can use SLAB. It's not that only one of them is intended for big iron, though. The only other selectable algorithm of importance is I/O scheduler. You can set the default at compile time, but it's always selectable at runtime. Plus each scheduler has runtime tunable integer variables. (There are also a few unimportant (in this case) selectable algorithms, e.g. choice of CPU frequency governor: ondemand vs. conservative vs. userspace.)