I find forks interesting because I'm interested in the way that open-source projects break down.
Martin Sustrik appears to be the lead author and architect of ZeroMQ, at least up until March 2010 [buyout, switch]. He apparently developed ZeroMQ as part of the open-source company FastMQ. Around October 2009, a Belgian company called iMatix bought FastMQ [buyout] and switched from its own messaging protocols to ZeroMQ [switch]. Pieter Hintjens was and still is the CEO of iMatix.
From around 2010, Martin Lucina was "co-maintainer" of ZeroMQ with Martin Sustrik [cv]. He and Sustrik both live in Bratislava [beer-mail]:
There were many discussions of process from at least August 2010 [workflow-proposal, releases, ego-per-git, first-fork, process1].
In at least one of these, Hintjens complains about Lucina's "ego", but asserts that ego is a major or the major motivation for work on ZeroMQ [first-fork]:
Hintjens wrote a tradmark policy for the ZeroMQ name in May 2011 [tm-email, tm]. A long thread developed from the initial announcement [tm-email]
In December 2011, there was a private email conversation between Hintjens, Sustrik and Lucina [metadict]. During this conversation, Lucina says that Sustrik "resigned as the benevolent dictator". Lucina [lagree] and Sustrik [sagree] agreed to release the contents of this private email conversation, but Hintjens did not [hdisagree].
Around January 2012 Hintjens proposed a radically open model of development [policy]. He had the view that "Maintainers are not developers and they have no opinion on patches". Thus maintainers should push all or nearly all patches after confirming basic process [policy].
In January 2012, Lucina wrote to the ZeroMQ mailing list [wtf] pointing out some of the more radical aspects of Hintjens' proposal. He gave the opinion that this way of working was already causing a decline in code quality and finished with an appeal:
From around 2010, Martin Lucina was "co-maintainer" of ZeroMQ with Martin Sustrik [cv]. He and Sustrik both live in Bratislava [beer-mail]:
Also, a lot of discussion between Martin Sustrik and myself got done in person, simply because we live and work in the same city (Bratislava, Slovakia).Lucina and Sustrik wrote an article about ZeroMQ in LWN [lwn].
There were many discussions of process from at least August 2010 [workflow-proposal, releases, ego-per-git, first-fork, process1].
In at least one of these, Hintjens complains about Lucina's "ego", but asserts that ego is a major or the major motivation for work on ZeroMQ [first-fork]:
I'll just point out that your own ego is dominant and not alwaysIn the same email he loses patience with one of Lucina's objections, points out that iMatix owns ZeroMQ, and suggests Lucina forks the code if he is unhappy [first-fork]:
pleasantly for others. I vaguely recall an original proposal for
multiple release branches that was quashed without sympathy (though
now you are arguing for exactly that). However without egos, and the
desire to dominate the work we do, this community would not exist, so
let's embrace that rather than deny it.
Shrug. You are free to fork any 0MQ repository and make your ownLuchina objected to proposals requiring pull requests on github, on the basis that it would lead to vendor lock-in [lucina-lockin], with some sympathy from Sustrik [sustrik-lockin].
versions and releases. It is LGPL licensed. The packages distributed
from the zeromq.org site, which iMatix owns, and labelled ZeroMQ, a
name that iMatix also owns, ultimately fall under iMatix's purvey. If
you feel iMatix has been a bad host to the 0MQ community, feel free to
fork. This is an essential freedom, don't hesitate if you think it's
necessary.
Hintjens wrote a tradmark policy for the ZeroMQ name in May 2011 [tm-email, tm]. A long thread developed from the initial announcement [tm-email]
In December 2011, there was a private email conversation between Hintjens, Sustrik and Lucina [metadict]. During this conversation, Lucina says that Sustrik "resigned as the benevolent dictator". Lucina [lagree] and Sustrik [sagree] agreed to release the contents of this private email conversation, but Hintjens did not [hdisagree].
Around January 2012 Hintjens proposed a radically open model of development [policy]. He had the view that "Maintainers are not developers and they have no opinion on patches". Thus maintainers should push all or nearly all patches after confirming basic process [policy].
In January 2012, Lucina wrote to the ZeroMQ mailing list [wtf] pointing out some of the more radical aspects of Hintjens' proposal. He gave the opinion that this way of working was already causing a decline in code quality and finished with an appeal:
Thoughts? Most of us here are software engineers by trade, surely you can see where this is leading.In an email on 4 February 2012, Lucina refers to a conversation with Hintjens the previous day in which Hintjens apparently asserted his leadership of the ZeroMQ community [metadict]. Lucina again appealed for support from other members of the community [metadict]:
If you care about this, the only way [4] to achieve change today is to beThis is footnote 4 from the same email:
vocal, and to criticise and argue those points of the process which you
care about. Having known Pieter personally for more than 10 years now, and spent many many hours arguing with him, I wish you luck!
[4] The other option is a fork (in the formal sense, not the Github sense). This is an option of *last resort*, and is not something to be taken lightly. However, it does put you on a fair playing ground; build your own community and process, and ultimately users will follow the leader with their feet.Later in the same thread Hintjens withdrew from further discussion with [hdisagree]:
I'm retiring from this thread now and will by necessity ignore anySustrik announced the release of Crossroads I/0 on March 15 2012 [fork]:
further discussion that isn't forward-focused and constructive.
Crossroads I/O is a fork of the ZeroMQ project.
While we acknowledge forking can be a painful process, we felt theThe Crossroads IO website gives some explanation of the trademark dispute:
ZeroMQ trademark policy to be overly restrictive.
Furthermore, the ZeroMQ community has also recently chosen to institute a light review process, which we feel is at odds with the technical quality and long-term goals we desire for the project.
The development policy for the new project is hardline Linux kernel style : all changes must be submitted to the mailing list as a patch [harddev]. They do not accept pull requests because [nopull]:To grow [a thriving commercial] ecosystem the project must be fully vendor neutral, and implement a liberal (e.g. Linux-style) trademark policy allowing use of the trademark for third party distributions of the software, as well as for plug-ins and extensions.
Pull requests can change while being reviewed. This makes it impossible to ensure that the code being merged is the same code that has been reviewed and discussed, which compromises integrity of the codebase.There are two interesting questions.
Pull requests are meant for delegation of work to sub-maintainers and require an established web of trust. We may consider moving to this model in future.
- Was there anything that the community could have done to avoid this fork?
- Which project will succeed?
[buyout] http://lists.zeromq.org/pipermail/zeromq-dev/2009-November/001362.html
[switch] http://lists.openamq.org/pipermail/openamq-dev/2010-March/001598.html
[beer-mail] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015768.html
[beer-mail] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015768.html
[policy] http://www.zeromq.org/docs:contributing
[cv] http://lucina.net/cv:lucina
[lwn] http://lwn.net/Articles/370307/
[workflow-proposal] http://lists.zeromq.org/pipermail/zeromq-dev/2010-August/004963.html
[releases] http://lists.zeromq.org/pipermail/zeromq-dev/2011-February/009220.html
[sustrik-lockin] http://lists.zeromq.org/pipermail/zeromq-dev/2011-February/009245.html
[ego-per-git] http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/009988.html
[first-fork] http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/009996.html
[process1] http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013986.html
[lucina-lockin] http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/014010.html
[wtf] http://lists.zeromq.org/pipermail/zeromq-dev/2012-January/015524.html
[metadict] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015744.html
[lagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015767.html
[hdisagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015774.html
[sagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015790.html
[tm] http://www.zeromq.org/docs:trademark-policy
[tm-email] http://lists.zeromq.org/pipermail/zeromq-dev/2011-May/011141.html
[fork] http://lists.zeromq.org/pipermail/zeromq-dev/2012-March/016429.html
[harddev] http://www.crossroads.io/dev:start
[nopull] http://www.crossroads.io/faq
[cv] http://lucina.net/cv:lucina
[lwn] http://lwn.net/Articles/370307/
[workflow-proposal] http://lists.zeromq.org/pipermail/zeromq-dev/2010-August/004963.html
[releases] http://lists.zeromq.org/pipermail/zeromq-dev/2011-February/009220.html
[sustrik-lockin] http://lists.zeromq.org/pipermail/zeromq-dev/2011-February/009245.html
[ego-per-git] http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/009988.html
[first-fork] http://lists.zeromq.org/pipermail/zeromq-dev/2011-March/009996.html
[process1] http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013986.html
[lucina-lockin] http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/014010.html
[wtf] http://lists.zeromq.org/pipermail/zeromq-dev/2012-January/015524.html
[metadict] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015744.html
[lagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015767.html
[hdisagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015774.html
[sagree] http://lists.zeromq.org/pipermail/zeromq-dev/2012-February/015790.html
[tm] http://www.zeromq.org/docs:trademark-policy
[tm-email] http://lists.zeromq.org/pipermail/zeromq-dev/2011-May/011141.html
[fork] http://lists.zeromq.org/pipermail/zeromq-dev/2012-March/016429.html
[harddev] http://www.crossroads.io/dev:start
[nopull] http://www.crossroads.io/faq
Wow, thanks for the history. It sounds like both sides have taken extreme views, neither of which is good. Regarding the no pull request policy, I wonder if they've ever heard of SHA1 hashes.
ReplyDeleteI'm curious if you've studied other forks. Is forking ever beneficial to a project?
The only other fork I've studied was NetBSD / OpenBSD. The emails from that fork are painful reading (http://zeus.theos.com/deraadt/coremail). It seems to me that the NetBSD "core" ejected Theo de Raadt in a remarkably pompous and self-righteous manner, and were thence incapable of any meaningful gesture of compromise. de Raadt forked NetBSD to OpenBSD, and has done other important stuff like OpenSSH.
DeleteI think that forking can be beneficial: for instance the Xfree/X.org fork actually moved the X project forward. Of course, the reason that the fork was necessary was because the project had come to a dead end.
ReplyDeletethanks Matthew, this certainly explains it much better than the one-liner wrt trademark. As far as your concluding questions: I think there's nothing the community can do when it finds itself at the peril of the DFL. I do hope the the fork succeeds (or both projects do). From a description of the reasons for the fork, it seems like ZMQ is racing e towards mediocrity, trying to be all things to all people, without worrying how it gets there.
ReplyDeleteMatthew, nice analysis. Some notes. I fully funded 0MQ (as FastMQ) for two years, while Sustrik led the business. At that stage it was failing badly. Clients did not like it. I asked Mato to help, he refused. As 80% owner, I folded it back into iMatix and decided that Sustrik's approach wasn't working.
ReplyDeleteTo his enormous credit, Sustrik rethought 0MQ as smart socket patterns. It's a serious achievement. It doesn't stand by itself though.
My views on development are strong but based on data; most of this comes from the 0MQ community itself but also from being part of way too many failing projects (open and closed) before.
Our trademark policy seems reasonable (have you read it?), and ironically, exists because Sustrik asked me to write it, and liked what I wrote. There was never any request to change it. It's a cheap one-liner to complain about it.
We can identify a number of conflicts; most are old and resonated through 0MQ for years. Look at 2011, when we had 2.1 stable, 3.0 unstable, 4.0 experimental, 3.1 beta. Most incompatible with each other. People did not know which way to go. Ask whether this was the result of a sane process. Or rather, which process was more sane - the pull request model by which I made stable 2.0 and 2.1 versions, or the push model that drove 3.0 and 4.0 (both of which were eventually trashed).
This is, for me, science, and that is valid data. The C4 policy isn't pulled from air, it's a collection of the best practices we've found working over the last years. Have you read it? Might be worth a glance. http://rfc.zeromq.org/spec:16
The problems C4 aims to solve are profound and often fatal to projects. How to prevent burnout of key people? How to avoid experts from rabbit-holing on the wrong problems? How to reduce the latency of change? How to make more accurate software?
Anyone who claims to know how to solve these problems any other way, please tell me! I've certainly never seen small expert groups succeed.
But after all this fork is going to be beneficial, once the doom sayers have had their lunch. It clarifies differences that always existed but which were toxic before. It creates space and reduces conflict. We need experimentation, and we need stability. The science of building software through communities needs to be challenged.
My bet is, and has been, on the community. The two Martins are betting on themselves. It's going to be fun to watch. At the very least the world has one more choice, and choice is always good.
Thanks for your thoughtful comment.
DeleteI did read your trademark policy and concluded that I did not understand the nature of the dispute. I assumed that I needed the arguments set out in more detail. I'll put a link to your policy in the main post.
As I was reading the posts I was struck by the fact that there seemed many proposals for process that could have been the basis for a compromise, at least on a technical level.
The Crossroads IO objection to the pull request review system [ http://www.crossroads.io/faq ] is that "Pull requests can change while being reviewed". Although this is true, it is easy to see when this has happened, and it is difficult to imagine that this feature would be used to push unwanted code. For this to succeed it requires the contributor to be incompetent or dishonest, and the maintainer to be unobservant. I assume the stated objections are not the full explanation for Lucina and Sustrik's preference for patches via the mailing list.
Have you tested your C4 proposal [ http://rfc.zeromq.org/spec:16 ] directly against the more traditional model of code review before push?