Ryan Gordon Brings Universal Binaries To Linux

[phoronix]

Phoronix: Ryan Gordon Brings Universal Binaries To Linux

One of the interesting features of Mac OS X is its "universal binaries" feature that allows a single binary file to run natively on both PowerPC and Intel x86 platforms. While this comes at a cost of a larger binary file, it's convenient on the end-user and on software vendors for distributing their applications...

http://www.phoronix.com/vr.php?view=NzYyNQ

[droidhacker]

Wow. You mean that our binaries aren't already big enough? I don't need PPC junk floating around my AMD64. I sure hope that fat-elfs can be reduced to skinny-elfs.

And for that matter, if it is going to waste the space anyways, why not just produce separated binaries to begin with? You know that whoever produces these fat-elfs will first have to produce skinny-elfs and then glue them together...

Seems to me that this is just an exercise in BLOAT. No thanks.

[yoshi314]

now he "only" needs to battle toolchain (mostly binutils) developers to officially include support for this new format. i don't think they like breaking standards and introducing new stuff too well (although "gold" linker was a surprise - it's still broken and in mainline)

one executable for >5 platforms - doesn't sound that good. i hope it will be possible to strip unneeded platforms from resulting file (if needed) without breaking it.

[mdmadph]

Disk space is incredibly cheap today -- I would forgive a bit bigger file if it "just worked" in Linux, and I think a lot of common users like me would agree.

[pvtcupcakes]

I think it would be better if you could just combine x86 and amd64 since that's what 99% of users use. Instead of having to include PPC and other obscure archs.

I like to develop software in my free time for Linux, but I only distribute the source code because I don't have the desire to compile for two different architectures.

[RealNC]

Bloated crap. No thanks. I hope this project fails.

[lordmozilla]

please no.

This is not a bad idea in theory, but I don't want anythign more bloated than currently. And this seems like a really easy way to push back X86_64 adoption and keep all these old ix86 packets floating about.

[Ex-Cyber]

Distributions no longer need to have separate downloads for various platforms. Given enough disc space, there's no reason you couldn't have one DVD .iso that installs an x86-64, x86, PowerPC, SPARC, and MIPS system, doing the right thing at boot time. You can remove all the confusing text from your website about "which installer is right for me?"Yeah, "given enough disc space". I'm sure that will be irresistible to Ubuntu, which fits a reasonably complete desktop install on one CD-ROM per arch/flavor by design. Or Fedora, whose install DVD images consume well over half of a DVD-5 per arch. Or Debian, which has multiple DVD-5 images per arch. Also, I could be mistaken (don't have time to look up the details right now) but I think that at least a couple of Debian's supported platforms have mutually incompatible requirements for bootable disc formatting.

I don't have any really strong feelings for or against this spec, but frankly that "benefit" seems pretty detached from reality.

I think it would be better if you could just combine x86 and amd64 since that's what 99% of users use. Instead of having to include PPC and other obscure archs.I don't see any reason to arbitrarily limit the spec to only two architectures, particularly considering the possibilities of ARM-based netbooks. Arch support for a given binary is entirely at the discretion of the distributor in any case.

[AHSauge]

Why is this bloated? Sure it will give you more than you need, but I will gladly accept this if it also includes x86_64 binary. Both my PCs run x86_64, but only one of them is x86_64 only. The other one has tones of 32bit libraries installed because proprietary software. I'm not going to speculate as for why they don't deliver 64bit version of their apps, but one thing I do know is that offering two sets of binaries for the same PC is confusing for everyone except technical people. If we want the Linux-platform to be more popular, we simply can't expect people, i.e your grandma and grandpa, to know whether to download and install the 32bit binary or the 64bit binary, because they don't know the difference and they don't want to know it either. What such people want is a binary that "just works". Universal binaries is just that.

please no.

This is not a bad idea in theory, but I don't want anythign more bloated than currently. And this seems like a really easy way to push back X86_64 adoption and keep all these old ix86 packets floating about.I disagree. Unless you really expect people to know the difference between 32bit and 64bit, this has to be done one way or the other to push forward the adoption to x86_64. As I said above, it's confusing to offer two binaries for the same PC, and we all know which one will work in any case.

[Remco]

Why is this bloated? Sure it will give you more than you need, but I will gladly accept this if it also includes x86_64 binary. Both my PCs run x86_64, but only one of them is x86_64 only. The other one has tones of 32bit libraries installed because proprietary software. I'm not going to speculate as for why they don't deliver 64bit version of their apps, but one thing I do know is that offering two sets of binaries for the same PC is confusing for everyone except technical people. If we want the Linux-platform to be more popular, we simply can't expect people, i.e your grandma and grandpa, to know whether to download and install the 32bit binary or the 64bit binary, because they don't know the difference and they don't want to know it either. What such people want is a binary that "just works". Universal binaries is just that.

How often do you install a plain binary for Linux apps? The correct way to distribute software is by non-executable installer packages. Those kinds of packages can provide any number of architectures in one single file. Or, if you don't like that bloat, discover the user's architecture from browser information and then offer the correct architecture-specific package by default.

[Apopas]

This could be very useful for commercial binaries, ie games and less confusing for the non experts. Ofcourse MacosX has an easier job since it supports just 2 archs.

[AHSauge]

How often do you install a plain binary for Linux apps? The correct way to distribute software is by non-executable installer packages. Those kinds of packages can provide any number of architectures in one single file. Or, if you don't like that bloat, discover the user's architecture from browser information and then offer the correct architecture-specific package by default.
Not that often, but that's not the point. The point is it does happen (games for instance), and you can't reliably discover the architecture via the browser. Sure, the most currently common browsers do provide it, but some (for instance Konqueror) don't, and what do you do then? Say "Hey! I don't know what CPU architecture you're running. It could be ARM, PowerPC, x86, x86_64 etc. Please download the appropriate version."? I don't think that's working too well.

And what do you mean by "The correct way to distribute software is by non-executable installer packages."? Do you mean the package manager? If so, you're only moving the problem to another area ("Which package manager does the user use?").

[kraftman]

Bloated crap. No thanks. I hope this project fails.

+1

PPC? Thanks, I don't use it and I probably never will.

[pvtcupcakes]

I don't see any reason to arbitrarily limit the spec to only two architectures, particularly considering the possibilities of ARM-based netbooks. Arch support for a given binary is entirely at the discretion of the distributor in any case.

What I meant was that the software developer/packager should be able to choose which arches to compile into your FatELF binary. If you only want to support x86/amd64 then you can do that. Or you could throw in PPC or sparc if you want.

[Apopas]

+1

PPC? Thanks, I don't use it and I probably never will.
Who cares for PPC except from few old-fashion Apple users? This binary format can be useful for binaries that support x86, x86_64 and maybe future arm platforms since I believe they will be in handy in the near future if Android succeeds finally ;)

[garytr24]

How often do you install a plain binary for Linux apps? The correct way to distribute software is by non-executable installer packages. Those kinds of packages can provide any number of architectures in one single file. Or, if you don't like that bloat, discover the user's architecture from browser information and then offer the correct architecture-specific package by default.

This is a very good idea for commercial software that doesn't necessarily want to play nice with .debs or rpm's or whatever. For instance, how about quakelive, the old loki games (I know the company's dead), world of goo?

A little bloat there is totally worth it if it increases probability that something just works.

I'd much rather have a bloated fatELF 64-bit native world of goo than just a 32-bit one plus all the extra libraries and incompatibilities I have to wade through to get it to work on a 64-bit system.

I don't think everything should use this, but some things definitely should as it would lower support costs for the developers, thereby encouraging them to spend more time actually making cool stuff. It's dumb to expect everyone to buy into your distro's package management philosophy.

[Adriano ML]

This sounds like an awesome idea. If it can happily resolve problems with kernel mismatch, dynamic linking, API/ABI and architecture differences... it may be a win for game developers wishing to support Linux and other *nix, maybe even macOS...


It obviously won`t fit the FOSS proposes and the way we manage package toda

[Remco]

Not that often, but that's not the point. The point is it does happen (games for instance), and you can't reliably discover the architecture via the browser. Sure, the most currently common browsers do provide it, but some (for instance Konqueror) don't, and what do you do then? Say "Hey! I don't know what CPU architecture you're running. It could be ARM, PowerPC, x86, x86_64 etc. Please download the appropriate version."? I don't think that's working too well.

I consider that a bug. Something similar was discovered and reported (and fixed) when the WineHQ website streamlined their download page to point to the right distribution package by default. Konqueror on Ubuntu didn't report Ubuntu. Now it does. Though this streamlining doesn't appear to be in use yet on WineHQ.
And what do you mean by "The correct way to distribute software is by non-executable installer packages."? Do you mean the package manager? If so, you're only moving the problem to another area ("Which package manager does the user use?").Non-executable packages are not the solution to this problem, but executable packages are a problem. They are impossible to secure, since they can run arbitrary code. A non-executable installer can be regulated through PolicyKit.

So executable installers are not desirable. Then this solution that packs a lot of architectures in one executable file doesn't solve the problem either.

The real solution remains a unified package manager interface. The upcoming PackageAPI solves most of the problem. What remains is a packaging format, installed by a PackageAPI-aware package manager, that every distribution agrees to support.

In the mean time, directing the user to the right package automatically works very well.

[garytr24]

This sounds like an awesome idea. If it can happily resolve problems with kernel mismatch, dynamic linking, API/ABI and architecture differences... it may be a win for game developers wishing to support Linux and other *nix, maybe even macOS...


It obviously won`t fit the FOSS proposes and the way we manage package toda

Idealism and practical considerations need to be and are always going to be in tension, yes? I'll go play my world of goo and scoff at the iceweasel folks now.

I think the prevailing thought in FSF guys minds is that they can force everything to be 'free', but I contend that lots of software won't exist if people can't make money off of it.

fatELF is a good thing for people that like to use their computer to do things. If there is something that can offer content management in secure way on a linux system, that would be a good next step, like a 'Steam' type service.

[vince]

I'm not sure how shared library loading is implemented in Linux, but if it maps binary into memory, does that mean we'll get a major increase in memory usage once FatELF goes mainstream?

[illissius]

Yeah, that's my main concern as well. Disk space is cheap, RAM less so.

[elanthis]

I'm not sure how shared library loading is implemented in Linux, but if it maps binary into memory, does that mean we'll get a major increase in memory usage once FatELF goes mainstream?

No.

Stuff that's used is loaded into memory. Stuff that's not is left on disk. Same way it always has.

[loonyphoenix]

This could be useful for "portable apps" and proprietary software. Do I understand it correctly that not only will these binaries work across different platforms, but also across different distributions? I don't care for the platforms very much, there are generally only two of them, but I do care that sometimes proprietary software is available as a package for only one or two distributions.

[Wyatt]

Sheesh, short-sighted cynics everywhere. I was more indignant that SDL switched to PulseAudio for its default sound, to be honest. Did any of you even read the rationale for this move? Really, I'd think at this point that at least Ryan Gordon, a well-known professional championing commercial games and development for our OS, has a good enough reputation in our circles that people would trust that he has good reasons for making a completely optional alternative specification for bundling multiple binaries together. Such as to ease development and user experience for people that don't know what arch they run.

Seriously, relying on web technology to reliably determine system architecture? Are you completely insane? Next thing, you'll be telling me that sidecar files are how semantic metadata should be transmitted. :rolleyes:

FatELF is actually a good idea, and I hope it sticks.

[pvtcupcakes]

This could be useful for "portable apps" and proprietary software. Do I understand it correctly that not only will these binaries work across different platforms, but also across different distributions? I don't care for the platforms very much, there are generally only two of them, but I do care that sometimes proprietary software is available as a package for only one or two distributions.

It's not a package; it's a binary. .deb and .rpm files are like tarballs that contain binaries. And binaries have always been cross distribution. (ELF binaries) This doesn't address the packaging issue.

[Remco]

What's wrong with PulseAudio?

[#kyz]

I really like the idea of bringing Universal Binaries to Linux. At least it gives those pesky commercial developers one less thing to complain about, which I think is a good thing for now. And hey, it could make it easier on new Linux users, which I have absolutely no complaints about.

I can see in a way though how some of you may not like it, however, it is entirely optional, if you don't want it in your project, simply don't use it. However if I was developing a project I'd at least give it a shot since I can't really say something bad about something I haven't even tried.

A game/app that comes to mind that could maybe benefit from it immediately is Nexuiz, it has x86 and x64 binaries in the same directory which may confuse the newer user. Of course I could be wrong.

[loonyphoenix]

It's not a package; it's a binary. .deb and .rpm files are like tarballs that contain binaries. And binaries have always been cross distribution. (ELF binaries) This doesn't address the packaging issue.

That's a pity. So then it isn't like "universal binaries" after all. As far as I know, those contain needed libraries, etc, being a package and a binary at the same time, no? (I never actually used Mac OS X :) )

[droidhacker]

Why is this bloated? Sure it will give you more than you need, but I will gladly accept this if it also includes x86_64 binary.
Your assumption is that they would include both binaries even if they were in one package, which it could be anyways.

The actual reason why certain proprietary software is packaged as 32bit only is that 32bit binaries work perfectly well on 64bit hardware, so there is no perceived NEED to do so. They would STILL have to compile it for both, which means that they will STILL not include both.

In fact, proprietary software tends to be distributed as runnable/installable archives. These same "install scripts" (like the ones distributed by AMD and NVIDIA for graphics drivers) *already can* include BOTH sets of binaries -- and select which ones to install based on the detected architecture. There is no need to actually combine the binaries! In fact, *AMD ALREADY DOES THIS*!!!

but one thing I do know is that offering two sets of binaries for the same PC is confusing for everyone except technical people.
How? If the software vendors followed AMD's lead and packaged them both together in the same script, it could *automatically* select the correct architecture.
If we want the Linux-platform to be more popular, we simply can't expect people, i.e your grandma and grandpa, to know whether to download and install the 32bit binary or the 64bit binary, because they don't know the difference and they don't want to know it either. What such people want is a binary that "just works". Universal binaries is just that.
What THEY need is for their software to be distributed in a magical system that automatically selects the correct package for their system, and at the same time, automatically installs all of the dependencies. Wait! We already have this... Its called PACKAGEKIT!

I disagree. Unless you really expect people to know the difference between 32bit and 64bit, this has to be done one way or the other to push forward the adoption to x86_64. As I said above, it's confusing to offer two binaries for the same PC, and we all know which one will work in any case.
As I have explained already, NO. This is NOT necessary. It is either up to the package vendor for their installers to automatically select the appropriate binaries for the system, or to the PACKAGE MANAGEMENT SYSTEM. There is *NO NEED* to bloat the binaries.

[droidhacker]

Not that often, but that's not the point. The point is it does happen (games for instance), and you can't reliably discover the architecture via the browser. Sure, the most currently common browsers do provide it, but some (for instance Konqueror) don't, and what do you do then? Say "Hey! I don't know what CPU architecture you're running. It could be ARM, PowerPC, x86, x86_64 etc. Please download the appropriate version."? I don't think that's working too well.

And what do you mean by "The correct way to distribute software is by non-executable installer packages."? Do you mean the package manager? If so, you're only moving the problem to another area ("Which package manager does the user use?").

The package manager is up to the design principles of the DISTRIBUTION.

Do you want to know WHY game vendors won't use this...
Think of this;
If your game is 1GB and you have 1000 users, each of whom downloads it, you have 1000 GB of download.

Now you decide to support 5 architectures, suddenly, the game size goes up by 500 MB to add in all the extra binary data. Each of your 1000 users now downloads 1.5 GB, which means a total of 1500 GB. That means that your NETWORK (which you pay for) suddenly has to handle 50% more data!

But if you keep it down to 1 package per arch, then though you have 5 packages of 1 GB, each user STILL only downloads 1 GB.

Do YOU want to pay for the extra bandwidth? Do you want the game maker to pay for it? If they pay for it, then they WILL charge YOU the difference.

So to summarize, the effect of joined multi-arch binaries:
1) Wasted disk space.
2) Wasted network bandwidth.
And that is *ALL*.

[droidhacker]

Who cares for PPC except from few old-fashion Apple users? This binary format can be useful for binaries that support x86, x86_64 and maybe future arm platforms since I believe they will be in handy in the near future if Android succeeds finally ;)

If?
Finally?
Its already blowing the top off of palm, and making apple sweat bullets... AND, the fun has barely *just started*.

http://wiseandroid.com/NewsItem.aspx?category=News&path=October&itemid=14

Android: one year old today.

BTW: that list is *far* from accurate (its got a few dupes, a few DOA, and there's a slew of things missing), but does give the idea.

[Svartalf]

+1

PPC? Thanks, I don't use it and I probably never will.

Heh... Think X86/X86_64/ARM and you might be closer to what we're likely to see. Having said this, I'm unsure that he'll get people to sign off on them. It'd make delivery a bit easier for commercial devs- but it'd be a nightmare in terms of testing the whole mess.

[Svartalf]

If?
Finally?
Its already blowing the top off of palm, and making apple sweat bullets... AND, the fun has barely *just started*.


No kidding. The fun is about to begin in about one month's time.

It remains to be seen if Verizon ditched at least part of their control fetish or not, but if the ads are close to the truth on the Droid site, they will have ditched it enough to make it compelling to code for that platform- however, I doubt that FatELF binaries, if they did take off, would be something you'd find in the AppStore there... ;)

[droidhacker]

for people that don't know what arch they run.
Huh? When all else fails, read the label on the front of the INSTALL DISK. DUH!

And this is linux,
You could always ask the user to run 'arch'.

For that matter, due to the propensity for downloadable installers, how about this; use a small installer frontend that does something like this;
#!/bin/bash
...
...
wget http://whatever/download/getfile?arch=`arch` -O installfile
...
#extract and install that file that was just downloaded.


Yeah, I know, you like it to download the WHOLE THING, so for *advanced* users, add in a "-d" parameter that just downloads the file, and a "-l installfile" parameter that installs a previously downloaded install file.

[Vadi]

Wonderful projects for the human Linux user.

Expert users compile everything on their own anyway.

[alazyworkaholic]

I'm happy to see this. Storage is cheap & the value of the convenience of having a single file can't be underestimated. I have a netbook (32 bit only) & will probably start using 64 on my desktop with karmic. This makes it necessary for me to keep two sets of files around when a package isn't available from the repositories. (acetone iso manager, frostwire, & a few others). lf I understood correctly & adding another arch multiplies the original size by ~1.2 or 1.3 rather than doubling it. It would be wonderful to be able to download files & updates to one computer to then pass them to another over a home nework instead of having to download two completely different sets. (I have a 100 kbps connection so you can imagine how much fun it is leaving the desktop on all night to update or download a few packages, then the netbook on all the next night! By the way, anyone seen a how to about downloading updates & packages first from one local computer & then look to the Internet repos to conserve my precious bandwidth?) Basically convenience is valuble & tar vs deb vs rpm vs run vs ... scares noobs.

[kraftman]

Heh... Think X86/X86_64/ARM and you might be closer to what we're likely to see.

This sounds much more interesting ;)

[larva]

I don't understand what the fuss is. I've noticed that the size of most of my installed packages are not dominated by the size of the binary. Even multiplying it 2-3 times wouldn't change that.

The fact is the primary benefits aren't designed for distros. Many distros have long since figured out how to handle multiple archs. They don't need FatELF.

FatELF is for two kinds of people: Proprietary software developers and naieve end-users. Despite sentiments expressed by the ideological fringes, these are valuable people.

FatELF toolchains would simplify proprietary development. Small and large proprietary software developers simply cannot sustain funding for the resources to match what the community has. So they tend to focus, as Ryan explains, on infrequent releases for small numbers of architectures and distributions.

I'm sure end-users would love to be able download a single Flash/whatnot plugin binary rather than have to wander through confusing web UIs. Video codecs are also a pain. Also, with FatELF it may someday be possible to simply copy a program from one machine to another without having to care about the arch (needs OS X style bundles though).

If space is really a priority the "bloat" can be stripped thanks to the simplicity of the FatELF format. It doesn't matter if it's stripped before packaging or after package installation.

FatELF can benefit non-proprietary developers too, but the benefits aren't as compelling. Assuming toolchains evolve sufficiently, it may be possible to avoid crosstool and have a single toolchain capable of building FatELF binaries for all arches. Then there may be opportunities to consolidate formerly-separate build, package, and distribution mirroring systems.

Also, the binaries may compress very well. Given that they probably share most of their strings and, if the data layouts are the same or similar, then the data segments might also have alot of common byte sequences. Text sections probably wouldn't share the same byte sequence distribution so wouldn't compress as well. Unless inlining is overused or debugging has been selected, my impression is the text sections tend to be pretty small.

Finally, even uncompressed and with multiple archs, the ELF binary for a given arch looks like its contiguous and aligned to a page boundary. This means only one arch of the FatELF needs to be mapped and the rest can be ignored -- so the size of the binary read from disk is virtually the same. The biggest cost is the potential addition of a disk seek. To avoid that I suspect it's possible to reorder the FatELF binaries (if not strip them) when the package is installed.

So to me it doesn't make much sense to complain about FatELF. If you want to spend your time complaining why not stick to a.out? ;)

[drag]

Yes... people complaining about bloat are just not understanding what is going on.

Having fat binaries means that you add a few megs onto major programs, at most. The bulk of modern software is going to be images, documentation, and other things like that. The actual machine code is going to be relatively small.

So the only thing that it affects is download time and disk usage.. and that is only going to be a fraction of the total size.

This means that if you download a game, browser, or plugin, it should "just run" regardless of the CPU your actually using...

[Ant P.]

90% of the replies here are from people who obviously didn't RTFA.