yearly release cycle

[Anatoli]

Hi Ricardo,

I find interesting some ideas in the proposed changes, especially I like
this: "uniform substring like “unstable”" (with startup-time warnings if
they change or become stable) and "Maintenance releases should be
no-brainers to install... This means that once you’re on 3.4.0, you can
always upgrade within the 3.4 series with a minimum risk. It also means
you get no optimizations, features, and the like.", though I wouldn't
limit the fixes to only the critical ones. Any bugfix that doesn't
change the behavior is included in the stable maintenance releases.

But I couldn't understand from the description what are the benefits of
tying major releases to certain calendar dates vs to make a release when
certain desired features are implemented and well tested.

What happens if some major new feature, that is a must for a new major
release to be published in a week, just isn't stable enough yet? Would
it have to wait for an entire year to be included in the next major
release? Or would you release it anyway as a stable release with known
issues?

Then, when you implement a new large feature, who would test it? Today,
for example, I (as an advanced user and a potential community dev) can
run 3.1 branch at some semi-production deployments (and I sometimes do)
and report issues. If, with the new scheme, you only guarantee that the
unstable branch just compiles, certainly I wouldn't be using it
anywhere, and probably neither would other users. Then pre-production
testing of new features would be exclusively the developers' task, with
obvious limitations.

So when the devs are sure that a new feature works well (in their setups
and for their use cases), it is included in the next major stable
release... and suddenly a lot of migrating users start finding issues.
That could create an impression that the new stable releases of Cyrus
are not that stable at all.


> As feature X is delayed (for various reasons that we can’t easily
> eliminate), it doesn’t just delay the feature, but also all the other
> minor bugfixes and optimizations that we’ve made in the master branch.

Why would a new feature of a stable release (3.2.0) delay bugfixes in
the current stable branch (3.0)?

> Also, it sets up the idea that we delay releases for the sake of
> fixes, instead of releasing the fixes that are ready.

I don't understand what you mean here, but with the current scheme
(AFAIK) the bug fixes go to the current stable branch (3.0) and all
users receive them without delays. New development happens in dev. When
some new feature is stable according to the devs (well tested in all
environments available to them), it is published as a new minor release
in unstable branch (3.1.x). This is expected to be fully-functional
releases, just not proven by the time and the community to be bugs-free.

I'm not sure how new major releases are managed today, but it could be
done this way: at some point in time, when devs decide that the
unstable/3.1 branch has accumulated enough features to be published, 3.1
is frozen for new features and it becomes 3.2.0-RC1 so the community in
general could start testing the new candidate stable version in their
test deployments. If issues are found, they are fixed in RC2, RC3, and
so on until no issues are reported for, say, 1 month. Then, the last
issues-free RC becomes 3.2.0 release.

At the same time, when 3.1 is frozen for new features, a 3.3 branch is
created and new features start landing there.

And the current stable branch 3.0 receives bug fixes as usual during all
this time. New optimizations probably won't be included in the 3.0.x
maintenance versions, but that's OK IMO. It's stable, not cutting-edge
after all. But it is bugs-free to the extent possible. All bugs, major
and minor (without behavior changes), are fixed there immediately.

This is a typical release cycle of many server projects. The main
advantage over date-bound releases is that the releases are published
when they are ready, not when we reach some specific point in time.

The disadvantage of the potential for delays could be mitigated by
defining certain criteria for the features to be included in each major
release. Also, some flexible dates could be defined, e.g. to publish a
major release every 6-12 months.

Regards,
Anatoli

On 13/12/19 11:59, Ricardo Signes wrote:
> Hey, remember last month when I asked about releasing Cyrus v3.2
> <https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2019-November/004509.html>?
> 
> That thread had some more conversation about what needs to get done
> before v3.2, and I wanted to come back to it and turn some things on
> their head.
> 
> Right now, we’re talking about Cyrus releases being feature-bound.
> “We’ll release v3.2 when feature X is done.” I think we’re not being
> well-served by that. As feature X is delayed (for various reasons that
> we can’t easily eliminate), it doesn’t just delay the feature, but also
> all the other minor bugfixes and optimizations that we’ve made in the
> master branch. Also, it sets up the idea that we delay releases for the
> sake of fixes, instead of releasing the fixes that are ready.
> 
> That is: every additional criteria for a new release is another doorway
> to delay. Instead of opening those doors, I would rather try to
> eliminate all of them.
> 
> I propose that instead of tying releases to milestones, we tie them to
> the calendar. For the sake of full disclosure: I am modeling this
> suggestion on the release cycle of perl
> <https://metacpan.org/pod/perlpolicy>, which I ran for several years. I
> found the process more than satisfactory, then.
> 
>  1.
> 
>     A new /unstable release/ of Cyrus is made every month. We promise
>     only that it compiled and passed the Cassandane test suite on the
>     release manager’s computer. It might contain regressions from
>     previous unstable releases, it might have crashers or corruptors. We
>     try to avoid any of these, but the goal here is a snapshot for easy
>     month-to-month testing. These are the odd-middle-digit releases. (3.3.x)
> 
>  2.
> 
>     A new /major release/ of Cyrus is made every year. We will have
>     tested it on as many configurations as we can readily test. We will
>     have, some time before the release, frozen the branch for risky
>     changes, to reduce churn. In the meantime, new work lives in feature
>     branches. (The changelogs from each unstable release provide a good
>     basis for the whole-year changelog!) These are the even-middle-digit
>     third-digit-zero releases. (3.4.0)
> 
>  3.
> 
>     A new /maintenance release/ of Cyrus is made for the last two stable
>     releases when there are enough fixes to critical bugs to warrant it.
>     These are the even-middle-digit third-digit-nonzero releases (3.4.1)
> 
> For the above to work, some more properties need to be maintained.
> 
> Maintenance releases should be no-brainers to install, so they must only
> fix regressions, crashers, security vulnerabilities, and the like. This
> means that once you’re on 3.4.0, you can always upgrade within the 3.4
> series with a minimum risk. It also means you get no optimizations,
> features, and the like.
> 
> Major releases must clearly document any incompatible changes or upgrade
> steps required. Because non-regression bugfixes aren’t backported, we
> want everyone to be able to upgrade from major release to major release,
> so incompatible changes must be kept to a minimum.
> 
> In part, this is just “don’t kill off a feature people use just because
> it’s a little annoying.” The more important one is “don’t introduce
> half-baked things that might need to change,” because people will come
> to rely on them before you get the updates finished. For features that
> will require multiple years to get right, they have to go behind a
> default-off configuration option. I’d strongly suggest they all have a
> uniform substring like “unstable”. That way, when a complaint comes in
> that the behavior of JMAP calendaring has changed, we can reply, “well,
> to use it, you had to turn on the unstable_jmap_calendaring” option.
> 
> If we go with this policy, we’ll need to…
> 
>  1.
> 
>     identify what issues are /blockers/ to v3.2.0, meaning they’re
>     regressions from v3.0 and would reasonably prevent someone from
>     upgrading; this does /not/ include all known bugs, since they may be
>     bugs that already exist in the last stable release!
> 
>  2.
> 
>     pick a release target for v3.2.0; I will arbitrarily suggest March 2
>     as “not too far off, but far off enough that we can get things in
>     order”; also, if you’re American, March 2 is 3/2 ;-)
> 
>  3.
> 
>     produce a changleog, and especially identify what changes in master
>     need documentation as “incompatible changes”
> 
>  4.
> 
>     produce a list of changes in master that should be put behind an
>     unstable configuration option and then do it
> 
>  5.
> 
>     decide when to stop merging non-release-related things to master
> 
>  6.
> 
>     make a plan for who will do monthly snapshot releases
> 
> I’ve spoken with ellie and Bron about just a few of these, such that I
> don’t think it’s all crazy. (ellie notes, correctly, I think, that the
> first set of releases like this will be the hard ones, where we work out
> things like “how do we keep track of incompatibilities, upgrade steps,
> and also how do we make snapshots dead easy to release.”) If there’s
> general agreement, I am definitely ready to pitch in and help try to
> make it work!
> 
> —
> rjbs
> 
>