Kanban vs. Scrum – By the numbers

As I’ve written earlier, we ended 2013 with with meaningful progress on our product development front. We could churn out new product capabilities in ~ 10 days from start to finish, and this gave our business a much needed boost of responsiveness. Thanks to our development decisions and sequencing of product priorities, we were delivering increasing value to the customer with the same work mass. This had significant top line impact on the growth of our business. Throughout 2013 and in January 2014, our 10 member development team followed Scrum as the development methodology.

How did Scrum fail us?

While Scrum produced great outcomes, it consistently resulted in undesirable artifacts

• We would under-deliver on our Sprint Commitment by 10-20%, which included unfinished work. At the same time, we would work on backlog items anywhere from 5-10%, and inevitably deliver some of it.
• Most of our releases would cluster towards the end of the 2-week Sprint cycle causing some stress on our systems which exacerbated the stress on our team.
• Most of our scoping decisions were made under time constraint of the Sprint deadline.
• We were unable to understand risk to delivery in any meaningful manner. This meant that outcomes could not be changed while the events unfolded.
• Product backlog was prioritized only every 2 weeks, though invariably there was some thrash as we deliberately traded commitments with backlog items.

The most undesirable consequence was the toll this methodology took on our team that was proud and ambitious. Each sprint left us with a feeling that we were under-performing. No amount of long term data could overcome the feeling of not hitting Commitment.

It’s not about Commitment, it’s about commitment

After much contemplation & reading, we decided that we needed to change our development process. We wanted a process that reflected our commitment to deliver value to our customers, not to Story Points (which are a surrogate for value). We needed a development process that would deliver the following

• Ability to prioritize in real time to constantly drive engagement & revenue, not in 2 week cycles
• Ability to make scoping decisions made under collective judgment, not time pressure
• Ability to deliver quality code — systematically
• Ability to respond to risk in a timely manner such that outcomes could be changed

We decided to give Kanban a try starting February. We created some swimlanes, some prescriptive limits max tickets allowed in each lane, and we were off to the races.

Evaluating Kanban

After a month of Kanban, all of us were feeling good about our development process, but was it working for us? I decided to inspect our development data. Before cracking open Excel, I stated the following null hypotheses which if borne out by data would prove that Kanban was no better than Scrum, and in fact worse in some ways

• Development cycle times remained as variable as they were during Scrum
• We were delivering fewer story points under Kanban
• Coding was no more parallelized as it was during Scrum

The resulting data is clear. In every category that matters, there was meaningful improvement.

 

In fact, the very shape of our development data shifted, evidenced best by these two charts which compared elapsed days or cycle time (a measure of our responsiveness) and parallelization (a measure of our teamwork and code quality), for a given work mass (denoted by size of the circle). Under Scrum, cycle time for higher mass items was higher than lower mass items. We were unable to parallelize effectively to reduce cycle time and increase our responsiveness.

Under Kanban, greater number of tickets were parallelized to a greater extent while cycle time was brought down systematically. You can see this by studying the spread of work across developers, while cycle time is contained.

Of course, there were some other important wins, which were a natural consequence of adopting Kanban. Work was prioritized in real time as needed by the business. This led to some great decisions. To cite an example, our product engagement doubled in 1 month, thanks to a new capability we had delivered. Our team was able to focus attention on making that product more reliable to support an increasing number of users, which resulted in more value for our users. Due to reduced work in flight, our product and development leadership was intimately involved in scoping decisions due to reduced work in flight. Most importantly, the team was a significantly less stressed as everything simply flowed – releases, priorities, code reviews. Of course, we learnt of risk to delivery objectively based on elapsed days, and this allowed the team to organize effectively.

So which is better – Scrum or Kanban

It is clear that for us, Kanban is a better development methodology – closer to a state of Zen compared to the start – stop world of Sprint. It is working for us (so far) because of some key reasons. Most important reason is our team cohesion, and a true sense of commitment our development team has for the business. In return, the business trusts and empowers the product team. This means when I tell our sales team or customer success team or our senior leadership team that our priorities can change any day, they react with celebration – not mistrust.