Table of contents
This page outlines the issues which faced the engineering team during the Lyons release. If you have any feedback on any ways for us to improve our process, please leave a comment below! We are also very interested in improving our process for the open source community as well.
The Lyons open source postmortem was held on 2009.04.01 with Arne, Guerric, Jessica, Max, Pete, Roy, and Steve to discuss the Lyons release - both the things that we did well and our shortcomings.
First off, we discussed the conclusions from the Kilen Woods postmortem and did a recap to see how we improved there.
Review of postmortem issues from Kilen Woods
We have adopted the SMILEYSCALE™ rating to judge how well we accomplished each task. Successful items get 
, failed items get 
, and mixed items get a 
. The overall SMILEYSCALE rating from the Kilen Woods postmortem are 5 
, 4 
, and 1 
.
Improve testing procedure
In Kilen Woods, we made good progress in expanding the role of automated unit tests in the API. However, we have turned off our nightly unit tests because of the false positives - somebody must go in and fix those unit tests. Without automatic binary building, test coverage on API code was spotty - with our new upcoming weekly binarie builds, this problem should be alleviated. There were no improvements on the UI testing front, and that remains a weak point in our test coverage.
DekiScript is also not receiving test coverage, and Dream receives only sporadic coverage.
For the commercial release, we did a great job of testing the commercial modules (thanks to Bob and Corey). Our incremental release (8.08.2) was quite sturdy, and had no known regressions. (8.08.2 was probably the most stable build ever released by MindTouch).
Going forward, we must improve our test coverage - this will be an ongoing struggle, as developers (?) will responsible for writing unit tests for their own features in the absence of a dedicated STE. We must also get our automated test coverage back on track.
Score:
Be wary of writeable resources
With the addition of site properties, the front-end no longer stores UI configuration files like the FCKeditor config on disk - instead we store it all inside the API. Yay! The development team has also been good at communicating changes the filesystem directory to the packaging team for RPM building. Successes all around!
Score:
Communicating open-source versus commercial
Let's not even touch this one.
Score:
Shipping delay
In Kilen Woods, we had a three week shipping delay as we tried to time the commercial and open source releases at the same time. For Lyons, we partioned the open source and commercial releases separately so we didn't have to deal with that overhead. However, Lyons still shipped a few weeks behind schedule, due to the risky nature of the update scripts (and the extra time we took to vet them).
Conclusion: We had a delay, but not for the same reason as Kilen Woods.
Score:
SVN syncing to SF.net
Less applicable now that we don't do a lot of branching. If we do branching, it would be a good idea not to do it in the public branch.
Score:
Localization
Translation tool is good enough, but could use more work. Our outreach and timing of the localization issues was not as good as it could have been - Guerric is setting up a listserv to directly contact translators for Deki.
Score:
Binaries from buildbot
The buildbot was a huge success - while we haven't done nightly bins yet, we are doing weekly bins that will allow users to update regularly. The binaries themselves contain information about the branch they were generated from as well as the revision number for identification through the API.
Score:
Blogging
We have kicked ass on this front - we have created a whole new home for the developer blog, and have been actively blogging about all activities related to Lyons. This has enabled us to gain valuable QA out of our community, and also has led to a high level of adoption - we've already had 120 upgrades to Lyons (without even publicizing it anywhere besides the blogs/forums!) less than a week after release.
Score:
Keeping installation documentation current
Fail. This has to be a top priority for clean up on the Developer Center.
Changing our release strategy
We've been good about keeping to the release strategy - the incremental releases went well with little developer overhead. We still have clearly demarcated major/minor releases. It will remain important for us to continue to communicate to our engineering team the process for development.
Score:
Postmortem issues from Lyons
Overall, we felt we shipped a fantastic release - the release was packed with tons of features which were a good mix work of point features as well as architectural changes to build off in the future.
Individual Notes
Pete:
- Rewrite rules sucked
- Services ... getting settings from instance to DW is hairy
- GOOD: update scripts
- Should involve MSI earlier in release cycle
- Not doing properties + lucene
- email notifications settings
- GOOD: notifications feature
Steve:
- Have more flexibility as an API team to use "experimental" features and not create UI features for each release to allow for experimentation
- Weren't able as a team to expose UI features ... need more use case driven designs
- GOOD: lots of stuff to write about!
Max:
- GOOD: Previews incredibly helpful!
- Send some of those kick-ass posters to community members
- Integrate dev blog more into dev center
- Spec gathering - needs more internal/community overview - many of the mistakes which led to the long release cycle were tied directly to the lack of rigorous overview of specs
- GOOD: Permission dialog
Arne:
- GOOD: Notifications
- Easy to build apps - proof of concept apps built in ~week!
- Learning curve with product was difficult
- Had to deal with existing pain points (decoupled services) - will be polished in time since it's all a learning curve
- No story for granular storage
- Spent too much time in the plumbing of notifications and didn't spend a lot of time on the email feature
Guerric:
- Shifting spec
- UI crunch (too much, too little time towards end)
- Continue to focus releases
Jessica:
- Themed releases/shorter release cycles
Summary
Release Shipping Delays
The biggest flaw from the Lyons release was abandoning our shorter release cycles in favor of a longer release cycle. While the intent was to give more time to architect bigger changes inside the product, there were too many big changes at once which caused massive slippage of the release schedule - we bit off far more than we could chew.
The longer the feature development cycle, the longer the testing cycle. Add in the fact that one of the features (attachments rewrite) contained many schema change, and the release schedule had to err on the side of caution (and thus led to its delays).
Going into the future, we will scope out releases into 8-10 week targets - this will allow us to focus on a few core features which also improves marketing/messaging around the release.
Specifications Gathering
Part of what led to the long release schedule was the vagueness in which we did spec gathering, and not having a final stakeholder who approved the specifications. While Lyons was the first release we publicly posted specs during the requirements gathering phase, we did not do a good job internally of following through with them.
Our specification gathering process should be more rigidly defined, with clear stakeholders at each level. Furthermore, each spec should be written with both UI as well as API input - both should collaborate on each specification.
For the next release, we will define a set specifications gathering time with community feedback - both the API and the UI team will collaborate on each specification together under the guidance of a product architect. At the end of the time period, the architect will sign off on the specifications, which will then be implemented.
We will also limit the number of specifications per release to avoid spreading the architect and the engineering team too thin.
Productization
The engineering team is now being migrated into more of a product team by factoring more directly in the requirements from other components of the company. More features in each release will be sliced to fall to fall into our product buckets - core/standard/enterprise.
The "M" release will attempt to introduce at least one feature that is geared towards this bucket.
Along with writing the feature and shipping it, it will our responsibility to make sure we put together a feature highlight page which emphasizes why this feature matters.
To begin, we will catch-up with all of our previous commercial offerings which have not gotten enough attention and create feature highlight pages with videos: Desktop Suite, the SQL connectors, and the Salesforce integration.
Disconnect between teams
In terms of major releases, Lyons had the least polish of all major features. The final version of Lyons was what we originally intended as two separate releases - we had intended on shipping the rough API features in Lyons, and follow it up with UI polish Lyons+. However, due to the long release cycle, we ended up combining the work items, and did a poor job of polishing many features in Lyons. This is most evident in the tagging update - it was half-finished, before we ran out of time and development resources to complete the release.
While nailing down the specs to a limited number to allow for better product management will go a long way towards alleviating this problem, another important aspect is to more closely tie the UI and API teams. We made the mistake of splitting the teams along skill sets, and trying to slice feature development into distinct API and UI phases. This did not work well for us - as the hand-off on certain features (email notifications) was not as well executed - this left many features in a funky state of unpolishedness.
To fix this issue, both the API and UI teams will work closely together during both the specifications as well as development phases - the features we ship will receive the polish that previous releases were known for.
Other Postmortems
- Tag page (Edit tags)
- What links here
Files 3
| Images 3 | ||
|---|---|---|
 |  |  |