Kanban vs. Scrum – By the numbers

As I’ve written earlier, we ended 2013 with with meaningful progress on our product development front. We could churn out new product capabilities in ~ 10 days from start to finish, and this gave our business a much needed boost of responsiveness. Thanks to our development decisions and sequencing of product priorities, we were delivering increasing value to the customer with the same work mass. This had significant top line impact on the growth of our business. Throughout 2013 and in January 2014, our 10 member development team followed Scrum as the development methodology.

How did Scrum fail us?

While Scrum produced great outcomes, it consistently resulted in undesirable artifacts

• We would under-deliver on our Sprint Commitment by 10-20%, which included unfinished work. At the same time, we would work on backlog items anywhere from 5-10%, and inevitably deliver some of it.
• Most of our releases would cluster towards the end of the 2-week Sprint cycle causing some stress on our systems which exacerbated the stress on our team.
• Most of our scoping decisions were made under time constraint of the Sprint deadline.
• We were unable to understand risk to delivery in any meaningful manner. This meant that outcomes could not be changed while the events unfolded.
• Product backlog was prioritized only every 2 weeks, though invariably there was some thrash as we deliberately traded commitments with backlog items.

The most undesirable consequence was the toll this methodology took on our team that was proud and ambitious. Each sprint left us with a feeling that we were under-performing. No amount of long term data could overcome the feeling of not hitting Commitment.

It’s not about Commitment, it’s about commitment

After much contemplation & reading, we decided that we needed to change our development process. We wanted a process that reflected our commitment to deliver value to our customers, not to Story Points (which are a surrogate for value). We needed a development process that would deliver the following

• Ability to prioritize in real time to constantly drive engagement & revenue, not in 2 week cycles
• Ability to make scoping decisions made under collective judgment, not time pressure
• Ability to deliver quality code — systematically
• Ability to respond to risk in a timely manner such that outcomes could be changed

We decided to give Kanban a try starting February. We created some swimlanes, some prescriptive limits max tickets allowed in each lane, and we were off to the races.

Evaluating Kanban

After a month of Kanban, all of us were feeling good about our development process, but was it working for us? I decided to inspect our development data. Before cracking open Excel, I stated the following null hypotheses which if borne out by data would prove that Kanban was no better than Scrum, and in fact worse in some ways

• Development cycle times remained as variable as they were during Scrum
• We were delivering fewer story points under Kanban
• Coding was no more parallelized as it was during Scrum

The resulting data is clear. In every category that matters, there was meaningful improvement.

 

In fact, the very shape of our development data shifted, evidenced best by these two charts which compared elapsed days or cycle time (a measure of our responsiveness) and parallelization (a measure of our teamwork and code quality), for a given work mass (denoted by size of the circle). Under Scrum, cycle time for higher mass items was higher than lower mass items. We were unable to parallelize effectively to reduce cycle time and increase our responsiveness.

Under Kanban, greater number of tickets were parallelized to a greater extent while cycle time was brought down systematically. You can see this by studying the spread of work across developers, while cycle time is contained.

Of course, there were some other important wins, which were a natural consequence of adopting Kanban. Work was prioritized in real time as needed by the business. This led to some great decisions. To cite an example, our product engagement doubled in 1 month, thanks to a new capability we had delivered. Our team was able to focus attention on making that product more reliable to support an increasing number of users, which resulted in more value for our users. Due to reduced work in flight, our product and development leadership was intimately involved in scoping decisions due to reduced work in flight. Most importantly, the team was a significantly less stressed as everything simply flowed – releases, priorities, code reviews. Of course, we learnt of risk to delivery objectively based on elapsed days, and this allowed the team to organize effectively.

So which is better – Scrum or Kanban

It is clear that for us, Kanban is a better development methodology – closer to a state of Zen compared to the start – stop world of Sprint. It is working for us (so far) because of some key reasons. Most important reason is our team cohesion, and a true sense of commitment our development team has for the business. In return, the business trusts and empowers the product team. This means when I tell our sales team or customer success team or our senior leadership team that our priorities can change any day, they react with celebration – not mistrust.

Decentralizing Product Management

For any enterprise software company or startup, the ultimate measure of success is how much money it makes for its investors. Money is the best surrogate we have for value a company delivers to the world – and by extension, the impact it has on the world.
For a product company like ours, its the product that delivers this value. Hence, the goal of product management is to build products & features that customers value & are willing to pay for. Our customers derive value when they use our products to address important design problems, but the journey starts much earlier, starting with how they discover our product, their perception of the product pricing, their experience during the sales process, and most importantly – their experience after they have bought the product.

Development cycles are no longer a problem area

On the other side, in order to be successful, product management works with designers & software developers in defining and sequencing what needs to be built to deliver value.While the latter has received significant attention, and almost down to a science. Thanks in part to the rich literature on development, over the last year we have continued to drive down our time-to-production, which today stands at 5 days (now with much reduced variability since we moved away from Scrum & adopted Kanban). This means that any new feature that customers would value can be built & deployed in less than 5 days. A major contributor here has also been our focus on moving developers closer to customers via support rotations, frequent usability testing, and integration with engagement-tracking software like Totango which give us rich visibility into how the product is being used. We don’t ship any new feature without tracking commensurate module/action in Totango.

Time to revenue as a metric

While for software development teams Time to production metric an objective works well, one of the metrics to track for product management becomes Time to Revenue, and minimizing this time requires significant focus in ensuring that information flows from the edges of the business to product management, and the other way around, as quickly as possible.

Now unlike software developers and designers, their primary job is not to respond to product management’s needs. Many constituents of the business interact with the world, and their primary job ranges from marketing & selling the product to helping customers adopt the product in greater numbers, thereby making money for the business. This makes flow of information inherently frictional, and the business must organize to make this as frictionless as possible

In order to solve this problem, most companies have product managers who go & speak with customers, sales teams, marketing teams, and customer success teams to learn about the world outside, and in turn helps them do their job better by arming them with valuable information. This makes product managers an important constituent. This also implies that there is reduced burden on other business teams in deeply understanding the needs of product management.

Decentralizing product management

At Sefaira, we have taken a different approach. We believe that organizing our teams to maximize our surface area that interacts with the world, product management needs to be distributed at the edges, and product management responsibilities need to be embedded with the edges of the business. In order to be successful here, we need to ensure the teams are always aware of key things we are trying to learn from the world, and in turn be able to push this information to product management. Conversely, they need to be the voice of product management.

Product management responsibilities need to be embedded in the DNA of all business functions. This reduces product management’s responsibility to organizing incoming incoming information, look for the most compelling hypothesis and test them by devoting maximum development fire power at these hypothesis. In turn, product management leans on the business to evaluate the outcomes.

What does this mean for other functions?

For different stakeholders, I will outline key areas where they need to think like a product manager in addition to their core responsibilities.

Sales rep

• When a prospect ask about functionality that the product doesn’t have, learn if the prospect would value the functionality they are seeking. If yes, what problem would they solve with it?
• When a prospect is unwilling to buy at proposed price, learn how much they are willing to pay for the product
• When a prospect is unwilling to buy, learn what problems the product would need to solve for them for them to value it

In turn, they should be equipped to talk about the kinds of problems we are looking to solve in the upcoming months, and learn if prospects value solutions to those problems. A negative or a positive signal both unlock different kinds of investments based on product management’s judgment.

Marketing

• Build new content and drive awareness based on what the product team is building
• Create and spread the word on success stories
• Build a compelling set of use cases for the product

Customer success

• Learn of blockers to product adoption
• Recruit users for usability testing and user interviews
• Learn how latest features are being used & provide insight to product management

Once these expectations are clear, the organization needs to be instrumented such that information can flow freely. We intend to rely on a combination of Asana, Slack, & Totango to stay aligned here.

The benefits of such organization are immense. Due to inherent decentralization of product management, this can shorten cycle times significantly, and scale well as the business & product footprint grows without the need for hiring additional product managers, or impacting cycle times. This keeps our business responsive to customer needs and allows us to bridge the gap between problems and solutions rapidly.