yearly release cycle

[Ricardo Signes]

On Tue, Dec 17, 2019, at 12:58, Anatoli wrote:
> Hi Ricardo,

Hi!

> But I couldn't understand from the description what are the benefits of
> tying major releases to certain calendar dates vs to make a release when
> certain desired features are implemented and well tested.

By promising a new major release every year, you know, when your significant improvement to Cyrus is accepted, it will very likely be released within a year. Right now, users who added a major feature in 2017 are still waiting for a stable release. For example, Sieve duplicate detection was implemented in March 2017. I don't think we have a stable version that has this feature. If this had been a contribution from a potential repeat contributor, it's easy to imagine that they'd have given up in frustration, by now. (Good thing it was good ol' reliable Ken!)

The problem with "we will release when X" is ready is that X might not be ready in a year, meaning all the little things don't get released. Also, you can't shove those into maintenance releases, because the little things still can be destabilizing, so it's less likely to be no problem to just upgrade.

In the event that a cool new feature isn't quite ready a month before release, I would argue: yes, it has to wait another year. I think it will be pretty rare that this happens, though. If it comes up, of course, an exception to the rules could be discussed, but in my experience, it won't. These kind of features, in largely volunteer-staffed projects, are rarely good at sticking to a timeline.

> Then, when you implement a new large feature, who would test it?

1. new large features should have tests written for them, which should be run by developers and dedicated test runners
2. some people always like to run snapshot releases; I have often done coding on dev releases of languages, and some people will surely run their personal services on snapshots
3. feature authors write features so they can use them; this means they're also both motivated and likely to use them before they're declared ready for general release

I feel pretty strongly that #3 is the big test. We're almost always close to bleeding edge Cyrus at work, because we have tons of new features that we rely on since cyrus-imapd-3.0.0. We know that many, many of these have been heavily tested in the real world, and we want to declare them generally ready for use, and then be able to do the same regularly as we move forward.

> Today,
> for example, I (as an advanced user and a potential community dev) can
> run 3.1 branch at some semi-production deployments (and I sometimes do)
> and report issues. If, with the new scheme, you only guarantee that the
> unstable branch just compiles, certainly I wouldn't be using it
> anywhere, and probably neither would other users. Then pre-production
> testing of new features would be exclusively the developers' task, with
> obvious limitations.

I think you are seriously overestimating the kind of stability guarantee you get from a 3.1 release. It's really not much more than the proposed snapshot releases, but on a looser timetable. Mostly, we get our current feedback from master, rather than snapshots, because there are fewer known snapshot deployments. Deploying snapshots regularly will give more points where we're specifically asking for feedback. (Also, I guaranteed Cassandane tests would pass, which is a *far* stronger guarantee than compilation.)

My expectation is that in reality, the snapshots will be, at any given time, very close to what Fastmail is running in production, or at least in (real, used by real people for real mail) testing environments.

> So when the devs are sure that a new feature works well (in their setups
> and for their use cases), it is included in the next major stable
> release... and suddenly a lot of migrating users start finding issues.
> That could create an impression that the new stable releases of Cyrus
> are not that stable at all.

I expect these features will have been heavily tested over the course of the time between releases.

> I don't understand what you mean here, but with the current scheme
> (AFAIK) the bug fixes go to the current stable branch (3.0) and all
> users receive them without delays.

There are two kinds of bugfixes. Some are "there is an obvious regression or crasher." Others are "there has long been a bug that meant that SELECT would fail on mUTF-7 sequences containing three hyphens in a row, and I fixed it!" The intent here is to include only the first category in new maintenance releases, because that optimizes maint releases for stability, making them easy to install without fear. The other fixes are put into the next possible snapshot for inclusion in the next major release.

I think your major concerns are:

1. new features might languish for a longer time than needed to be known stable
2. snapshots will be less reliable under this regime than before

I feel strongly that #1 will not be the case. We can always talk about making an interim major release if it comes up, but I am predicting that it will not, and if it does, we will think about it and decide that the effort to make sure we feel good about an unexpected major release is not enough to push us to rush. I acknowledge that reasonable people can disagree on this, but the good thing is: we can wait and see!

I disagree that #2 will be the case. Master does not churn with very much untested code, and I'm hoping we will slow it down even further by putting more features into feature branches until they're more battle-tested. That will get us more "this snapshot introduces feature X, which has been tested by production users and load!" rather than "master has been growing feature X in pieces for months, and it's all a bit weird."

In general, with Fastmail probably-always running a fast-forward of a snapshot in testing, I feel pretty confident about snapshot use for similar under-load testing elsewhere.

-- 
rjbs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20191217/49952f19/attachment-0001.html>