Release Strategy

Major Releases

MindTouch averaged a major release every three months in 2008 (Itasca in February, Jay Cooke in May, and Kilen Woods in August).  This pace will slow down in the future - for the immediate future, we are targeting major releases every 4-6 months.

Major releases contain breaking changes, which may require configuration settings in your environment or update scripts to be run. APIs and interfaces may be deprecated between major releases; any major changes are documented in the release notes, so be wary if you have custom MindTouch code, as major releases may cause your code to break.

Major releases also receive a project name; MindTouch has traditionally used Minnesota State Parks as inspiration for release names.

Minor Releases

Minor releases are only bug and security fixes. Minor releases do not require any server configuration changes; they just require an updating of the MindTouch codebase. In the past, minor releases contained both features as well as bug fixes - this is no longer the case.

A minor release is scheduled 4 - 6 weeks after every major release, which coincides with the Enterprise release.

Step 1: Feature determination

Over the course of a release, our developers get a sense of new features for MindTouch through interaction with community members (as well as our own vision). The first thing we do is capture every single idea for a new release on a whiteboard; this process involves MindTouch developers, our customer support team, and our sales team. The list is captured online, and a public request for review of the new release is made. During this time, specs are drawn up and we try to get a scope of how long each feature would take to develop.

Features are then triaged into three: must haves, nice to haves, and for later. Once the must haves are determined, further attention is given to those specs, and a consensus is reached for the engineering team on how to accomplish each task.

This step usually overlaps with the "stabilization" period of the last major release; this serves the dual purpose of preventing developmental trunk from diverging too much from the stable branch, and keeping development cycles free in case critical issues arise.

This period usually takes about a month.

Step 2: Feature development

Engineers are tasked to individual features, and are set to work on each item. Must have items (regardless of how big or small they are) require attention first during the development cycle. Once all the must have items are done, developers are free to tackle nice to have features. Once the core must have features are done, a release date is set three weeks in advance of the final code freeze.

During this time, our QA team are tasked with drawing up a testing plan for changed features, and start executing.

The lenth of this particular part of a release is variable and subject to a lot of subjective decisions.

Step 3: Bug fixes and stabilization

Engineers are tasked with fixing bugs during this cycle. A couple weeks after the most critical bug fixes are fixed, a "code freeze" is issue, and a release candidate is posted. During the code freeze, only check-ins with bug reports are allowed into SVN. A release candidate is posted weekly afterwards until the release is considered stable, and the release is posted.

Step 4: Releasing

Release notes are drawn up, and the source release occurs. After the source release, the shipping SVN branch is updated for VM users, and then a new VM image is made. This is followed with updated RPM and MSI packages.

Wik.is is not part of the overall release strategy, due to the complexities of updating at scale.

Step 5: Post-mortem

Every release is accompanied with a post-mortem, where the engineering team discusses weaknesses in each release (be it features or process). These post-mortems are posted publicly for our community to review. After the post-mortem ... we go back to step 1!

Old Bugfix Strategy (before 8.08)

In the 8.05 family of releases, all bug fixes and new features were done in trunk - when we arbitrarily set a date for the incremental release (8.05.1), we would take the sum of all changes in trunk and merge them into the development stable 8.05 branches; these would then be merged into the public stable branch.

There are a couple of benefits for this approach:

• Developers don't face the difficulty of fixing the same bug on two separate branches (most developers did not have an environment configured for both trunk and the development stable
• We remove the cognitive burden of somebody (most likely a product manager) assessing the risk and validity of a filed bug ("Does this require a backport?") and determine if they require a backported fix.
• Testers are only required to perform regression testing on one branch. During the 8.05 release family, we lacked an API unit tester, so the whole application was tested by two QA guys who were primarily targeting front-end changes.
• Shipping releases is easier, since merges are the sum, and there are zero risks of conflicts.

The downsides of this approach:

• Requires more rigorous QA oversight - trunk must always be in a semi-stable shipping state if this approach is to work (which it did). 
• Huge architectural breaking changes are not possible - this approach forces developers to take a piecemeal "baby-step" approach towards changes
• This approach forces trunk to go into "code-freeze" for incremental releases (since the sum of things on trunk are what get shipped)
• Does not allow us to do specific "hot fixes" for a release - hot fixes must be applied to trunk, and the sum of changes must be applied. This makes patching zero-day vulnerabilities difficult - however, with our aggressive release schedule (a release a month), this problem was not readily debilitating.

Using this approach (which was in place since roughly the Itasca release), we were able to rapidly release multiple major and minor releases over a short period of time. This "micro-release" strategy was also in place during a nascent time for MindTouch's business growth, where flexibility of the development schedule was paramount. This flexibility allowed us to accommodate large projects like Mozilla and TopSan as well as the development of the "Enterprise" versions of MindTouch . 

Now that the difference between Enterprise and Open Source editions of MindTouch are clear, we decided to change our strategy to place an emphasis on creating the best product without placing a high degree of flexibility on the release schedule. These changes are being implemented for the 8.08 release and onwards.

New Bugfix Strategy (8.08 and onwards)

The new approach's paramount goal is to provide a consistent stable branch from which we can target bug fixes and security fixes for a release at any time. With the release of the Enterprise branch, it becomes critical to MindTouch to focus on stability instead of rapid releases for our customers and users. This shift coincides with our longer development cycles, which will inevitably lead to less visibility when scheduling releases and projects.

In the graphic above, the 8.08 family of releases outline the bugfix strategy. The two dots indicate two security bug fixes which are discovered at two points during the release cycle. Developers are responsible for fixing identified issues in both trunk and the development stable branch at the same time. Testers will be responsible for verifying bug fixes in both branches. At release time, we take the sum of all changes on the development stable branch, and apply them to the public branch.

There are a couple of benefits to this approach:

• Easier minor release management, as managing the scope of minor releases becomes easier (we no longer have to factor in features). This becomes critical as we continue to roll-out to more platforms; the delay between the source release and subsequent packages (MSIs, RPMs, VMs) maybe delayed. This also allows us to time a commercial release after the open source release which will contain bug fixes found during the extra stabilization period.
• Longer release cycles are acceptable - if a zero-day vulnerability is found, we can simply patch the development stable branch and ship a new minor release.

The downsides of this approach:

• Decisions about individual bugfixes must be made ahead of time (this becomes especially tricky as certain checkins won't actually be "bugfixes" but "feature improvements") - a strong product manager is required to be able to say "no"
• Overhead for QA and developers