I still don't understand
By glazou on Tuesday 12 July 2011, 11:44 - Mozilla - Permalink
I could have titled this article "I understand even less". Asa wrote the following in a comment on an article on Gerv's blog:
We have and we will break add-on API compatibility with every release, certainly binary add-on compat
I spent part of the night thinking about it but still, "Urgh..." is the only thing that comes immediately to my mind. I'm shocked, so shocked, you could not believe how shocked I am. Bumping version numbers every six weeks for strictly declarative (xul/js/css/xbl) add-ons is one thing and breaking binary compatibility for binary-based add-ons is another one. The former case is painful but doable in a lot of cases. I still think that add-on authors (I am an add-on author) will rapidly feel the pain but that's nothing in comparison with breaking binary compatibility every six weeks...
Some years ago, I contracted for Wengo, the VoIP division of a major telecom player here in France. We took their huge pile of binary libs, encapsulated them into binary components (painful but still simpler than js-ctypes) and I built both a SIP-based VoIP add-on for Firefox and a standalone application on that basis. Quite impressive, I must say. With a mandatory recurrent six weeks XPCOM component update/rebuild/reship process, my client would never have done that work because that's impossible to sustain.
On a more personal note, some of my personal add-ons to Firefox are binary-based, and there is no way I can dedicate time every six weeks to them. Most add-on authors work on add-ons on their personal time and will be in the same case.
Add-ons are the major plus Firefox and Thunderbird has over browser competitors. That ecosystem is unique. Harming that ecosystem is a counter-productive decision of such a magnitude I am completely lost these days reading Mozilla's strategic decisions. It seems to me the path taken is an engineer's path, not a market-based one. It seems to me based on a wrong analysis of the market, the competition, the users' needs and the marketable differentiating factors.
Our PR team and our Marketing team are happy with the current system and they're the ones on the front lines of "publicity". Also, that negative publicity has already happened. It's in the past. It may happen to a much lesser degree with 6 and probably even less with 7 and eventually it goes away as the version number disappearing act finishes
I think I see what's going on here. Asa is betting on the extinction of the criticisms over time. That's why despite of the incredibly negative press world-wide, we saw no serious reaction and why no improvement has been seriously discussed. The weekly meeting minutes of Moz are, from that perspective, incredible: almost no mention of the world-wide fuss Asa generated two weeks ago. First, that's certainly not how the community feels Mozilla should react. Gerv seems to agree (see comment starting with "That is one of the most depressing things I've seen you write in a long time"). This feels like an occasion long ago when Blake Ross was forced to "discuss in Bugzilla" something he had made a decision on, and responded by posting there one single word: "discuss."
At this point, and I never thought I could say that one day, I don't trust any more Mozilla's strategy for Firefox. I don't trust any more its Product Manager for Desktop Firefox. It's not the person, it's the strategy. I'm not angry, I'm incredibly sad. During all these years, I tried to draw the attention on markets addressable by Moz and not addressable by other browsers. I failed. During all these years, I did all my best to give my opinion about how to extend the ecosystem, the true cornerstone of Firefox's success. I succeeded implementing these ideas for my own product, BlueGriffon, but I failed for Firefox. I'm sorry, I'm so sorry.
At this point, I think the CEO of Mozilla should talk, loud and clear, and talk about this more with both the community and the users. Enough with corporate blah-blah, and I could say corporate bullshit.
Comments
Yes, Mozilla didn't communicate it clearly yet - but binary XPCOM components in extensions are a non-starter now. Firefox 4 didn't only change the way you declare components, it also introduced a version flag that will increase with each new Firefox version. In other words, you can no longer rely on the binary interfaces not changing, you are forced to recompile with the new SDK for each Firefox release. That's understandable given that binary compatibility was a major pain factor - but paired with rapid releases it makes binary XPCOM components unusable. There is no way anybody will recompile their XPCOM components every six weeks and ship extensions with at least two versions of these components for each platform.
So the way to go is js-ctypes now, no alternatives. You can ship a native library with your extension and have a JavaScript-based XPCOM component as a wrapper for that library. This should be the easiest way to adapt existing extensions with binary components.
Criticism fades as people leave the project or decide their voice is being ignored. Open source contributors (particularly add-on developers) aren't generally going to stick around and try to change things. They'll complain loudly, see how the wind blows and then walk to one of the many promising projects out there that aren't taking that path. They don't owe Fx anything. If Fx starts sinking (from their perspective) then they won't go down with the ship. They'll jump overboard, hoist themselves onto the good ship OtherProject and get on with their lives. OSS projects seldom die with a bang - but a whimper - as the community just evaporates transparently until suddenly the core developers think 'hey, where did all those people go?'. There's plenty of other boats in that particular sea. (This metaphor has been hung out to dry. On the masts of a ship.)
With another hat on: I'm currently on the Aurora branch.. however I can't upgrade to Fx 7 since none of my extensions are supported. As we've moved through the versions, the TLC of extension developers for their projects has disappeared and the time-to-release for their patches has skyrocketed. I've already given up working on add-ons myself because.. well, frankly, I have a life to worry about. Between reviews, bug fixing and keeping up to date I'd lose all my free time to maintaining them.
With yet another hat on: I work in an enterprise and influence decisions regarding browser adoption. I cannot recommend Firefox in any context, where before I most assuredly could (and from a position of strength). That's a big shame on so many levels. Not least because that's how I learnt Firefox's code base in the first place, through developing a browser for another enterprise. I assume (though I'm not around anymore) that they are either stuck with the last version I released or have ditched Fx in favour of another browser with a commercial focus. They've been left out to dry because they're not considered stakeholders but... why? Hell, the reason we needed to deploy a browser is we were targeting one of those lumbering behemoths that still use IE6. We were on the front-line of addressing the use of poor web technologies and now my former employer is kinda stuffed. In the future I'll be forced to say "don't use Firefox, you can't rely on it and the cost is too great. We need to work with Microsoft to get our clients using the latest IE, writing against IE and training for IE because that's actually easier" I don't want to be doing that.
In a way I understand both arguments, the need to land bigger updates more timely versus the need to fix bugs.
I cannot see why this cannot be combined in an approach that meets everyone half way. Have a tick, tick, tock approach. Do two minor updates, being security, optimisation and refinement related. Use this time to work on the “Tock” stuff, the improvements to underlying technology, infrastructure, the stuff you know that will have greater consequences and impact. This relieves pressure from add-on developers, main developers and is less confusing than the current mess.
Firefox have enough wrong with it to justify the two minor updates. Use these to polish the user experience and also to learn where the slowdowns in the underlying tech are. Memshrink is a great example of the type of work that can be done during Tick stages. Azure is the type of stuff that can be done for a Tock release. You keep on delivering updates every 6 weeks, but you do it in a way that does not kill your community or your marketability of your browser.
Browsers have matured; all the browser platforms are now about open specifications. All that matters are the user experience, and frankly, Firefox’s have a lot of work needed.
This whole number marching will get tiring too; News outlets will stop caring every six weeks about a new Firefox release, because there is nothing to report on.
We all would like to see Firefox succeed and resume its growth. For this to happen, you need to make some business decisions. They will not go down well with the loyal supporters, but guys, without their support, who are Mozilla? Just another company that turns out browsers that will eventually sell itself?
Asa has shown nothing but complete ignorance of late. He should be fired.
"It seems to me the path taken is an engineer's path, not a market-based one."
That's not an engineer's path. Engineers do the job right. Engineers pay attention to details like backwards compatibility, and plan ahead to maintain it for as long as possible. Engineers like the people who maintain the Linux kernel/userspace boundary/ABI, the glibc2 ABI, the GTK2 ABI, the zlib ABI, the Java ABI, the libx11 ABI, the lib{jpeg,png,gif} ABI, etc...
Failing to maintain compatibility every 6 weeks is the work of fucking cowboys.
The problem I have that (at least in the Linux world) Firefox is like IE ... you can always expect people to have it installed and it is hard to ask them to install something else.
We have a stable binary interface, if you will, and that's js-ctypes. But for XPCOM, we don't care if its binary interface breaks or not. In fact, we change its interface version for every version bump, just because we cannot and don't want to guarantee compatibility. You need to use js-ctypes if you want stable work with binary modules. That way you have a stable interface.
I half wonder if we could hook AMO up with a test-harness for addon authors, and the ability to compile extensions. If everything compiles and passes, you get bumped for the next release. I think I remember seeing ideas like that tossed around.
But yeah, that's a lot of work on AMOs end, and still expects addon authors to actually upload good code and tests (I'm guessing most binary addons aren't even distributed through AMO). And all to maintain something that we want to encourage addon authors NOT to use anymore because it is going to be MORE difficult for them in the long run.
@Thinus: Did they stop caring about Google Chrome releases?
@Karellen: Speaking of Linux, they sure care very much about binary modules, don't they? :->
"I refuse to even consider tying my hands over some binary-only module" — Linus
KaiRo, I don't know you anymore. You changed your behavior since you were hired by Mozilla for a full time job. Once you were always the first person shouting his doubts and asking for reasonable and logical answers. Instead now you look like a perfect yes-man soldier who repeats the sentences has learned like a tamed parrot. What happens to the good old KaiRo??? We want him back!!!
Karellen, we did the XPCOM compatibility thing for a decade or so. It's saddled us with a huge technical debt that now needs to be paid off. There were a bunch of incorrect initial decisions (some incorrect with benefit of hindsight, some just solving the wrong problems) that are now preventing us from implementing new web specifications correctly, not to mention hurting performance and memory usage.
Now we could keep on as we are, not implement web specs properly, and be bloated and slow. But then, why would anyone bother with that? I sure wouldn't.
Or we can take what we have learned and work on a better setup. Think past ABI changes in all the packages you have listed. It's happened. More than once for many of them.
Now if we're working on a better setup, we can do it in one of two ways. We could either go off on a branch for two to three years and not ship anything in the interim, then land it all at once. This does not sound like a path to success to me; that's what Netscape did in 1998, and it took nearly a decade to recover from that. Even if you posit that the right time to start counting is 2002 or so, it took 5-6 more years after that. The other option is to land the changes incrementally. That means that binary compat will be breaking in the meantime, especially binary compat of the sort that some extensions depend on (internal APIs like nsIContent and the like).
So at that point the only question is what frequency of breaking binary compat is ok. If we posit that releases do happen every 6 weeks, should it be OK to change nsIContent and other internal APIs on that timescale? Should it be OK to change nsIDOMNode (which we really need to do to implement changes to web specs!)? Other APIs?
I don't think anyone is _happy_ with the current situation, but we're forced into it by the enforced decade of freeze and stagnation and falling behind and by the lack of a separation between binary APIs for binary addons, web APIs, and internal Gecko APIs. We MUST be able to change these last two, and change them quickly as needed...
Daniel, see http://xulforge.com/blog/2011/07/ve...
"On AMO, we have implemented a system where we automatically detect which add-ons are compatible with the latest Aurora version (the version that is between 12 and 6 weeks from release) and then upgrade those that do. All developers for the tested add-ons should get an email explaining their add-on was upgraded, and if not, why. This has been fairly successful, and we have a high compatibility percentage (relative to usage) at the moment (it’s even higher if you ignore the .NET Assistant, which could be compatible now). However, there are still many popular add-ons that aren’t compatible, specially those hosted elsewhere. And the burden on those who develop add-ons with binary components is pretty much constant, because they’ll have to update their add-ons every 6 weeks, with no exception."
So this should only be a problem for binary addons (about 10-20% of all addons that blog post says), where the authors can switch to js-ctypes to get the same functionality.
Drivers/Modules are a part of Linux's internals. You can change your internals around as much as you like. The API you present to external developers is what must stay stable.
Linux succeeds. Programs compiled for Linux 2.0.0 will run on 2.6.39. (modulo libc changes, but you can install/copy libc5 binary libraries to a 2.6.39 system alongside libc6/glibc2) The external-facing API/ABI is stable.
Karellen, one problem is that historically most binary extensions suck and poke into internals. That means that we have the choice of not changing internals or forcing them to recompile or accepting that they will crash the browser all over the place after the update.
Karellen, oh, and your libc thing is exactly spot-on, except you're thinking about it wrong. The only reason things "work" is that people just intall multiple libc versions and run against a particular one, not because the libc ABI never changed. But people seem unwilling to do that with Gecko, for various reasons.
@Boris:
The issue is that you are forcing binary incompatibility even if there are no actual incompatible changes.
Have then been actual changes that break binary compatibility between 4/5/6/7/8?
It seems that Asa's (unconscious, let us hope) standpoint is: we don't want binary add-ns, f??? them, they have to convert to js-ctypes or die, just like we don't want enterprise users, f??? them, let them go back to IE which will kill them.
<b>Note:</b> This is caricature and black humor; but from a certain point of view it sure is the impression Asa's recent posts are giving.
Boris and mkaply: The change that breaks binary compatibility between Gecko 7 and Gecko 8, at least, is a one-line change by whgich the XPCOM version, which was 7, became 8. See:
https://bugzilla.mozilla.org/show_b...
http://hg.mozilla.org/mozilla-centr...
Michael, there have been about a dozen changes just to nsINode since Firefox 4. This interface has changed between 4 and 5, between 5 and 6, between 6 and 7, and between 7 going to aurora and today. So for example any binary addon that uses nsIContent or nsIDocument would need to be recompiled across each of those releases.
Or, if you say that people shouldn't be using those interfaces, note that nsIDOMNode changed from 4 to 5 and from 6 to 7. nsIDOMDocument changed from 4 to 5, from 5 to 6, from 6 to 7. Neither one of those has changed in the last 10 days, but they've got another 30 days until 8 branches from mozilla-central, and I know there are patches waiting for review that will be changing nsIDOMWindow for Gecko 8.
So yes, there have certainly been changes that break binary compatibility.
The point about libc compatibility is that, when moving between major versions, they've thought about it and explicitly provided a way to maintain compatibility for old programs.
Sometimes you do need to release a new major version and break the ABI. No-one can see indefinitely into the future and get everything right first time. I'm not suggesting that.
But libc has been written so that multiple major versions (libc4/libc5/libc6) can all be installed side-by-side simultaneously, in order to allow existing programs to work. (Similarly gtk/gtk2, qt2/qt3/qt4, etc...)
Does Mozilla allow for multiple versions of Firefox to be installed side-by-side, both co-existing simultanously, so a user can use a new version to get the most out of whizzy HTML5 sites (e.g. Google+) but keep an old version around for when they need to use old extensions?
No. That's the difference.
Plus, while I say ABI, I also meant the javascript API, as there is no API/ABI distinction for scripted environments. Because this isn't just about binary extensions, this is also about js-only extensions making use of publically documented, recommended, APIs, which are still, apparently, being broken.
Karellen, Firefox absolutely allows multiple versions to be installed side by side. I have Firefox 2, 3, 3.5, 3.6, 4, and 5 all installed on my machine right htis minute, using different profiles, and use them for testing all the time. With the newer ones you can even use Sync to have them effectively share profile data and the like.
Now we don't provide security updates for older versions. Neither does gtk, last I checked. I haven't looked into libc and qt.
Breakage to JS-facing APIs, especially recommended ones, is very rare; we try hard to not have it happen. Major architectural changes (e.g. e10s) can force it, of course.
I just book marked your blog on Digg and StumbleUpon.I enjoy reading your commentaries.