Being data driven as a product leader is considered to be table stakes today. However, my view is that with the advent of easy access to data, ease of building software, and clever marketing (Hi Big Data!), we have veered too far on this view. There are strong limitations to being data driven. I try to highlight three ways in which being data driven is a bad idea.
no data for behaviors that are latent
One of the most successful innovations in the history of our civilization was the decimal system. It gave us an easy way to count. It unlocked a latent capacity – which was our ability to count. However, if you were asked to bring data to prove how successful the decimal system was going to be, you probably won’t be able to find any systematic data on this.
Fast forward to a few years ago, if you look at contemporary examples of some of the most successful products, like the iPhone, all data suggested that there was no need for an expensive phone that didn’t have a keyboard. The reality is that game changing products are borne out of a vision of how the world ought to be, & the products themselves become a means that realize the vision.
Instead of collecting data prior to building a product, what’s important is to articulate what sort of data will be collected to determine if the product is realizing the desired vision.
Poor decisions based on easily accessible data
Let’s assume that you actually have data to prove that the product decision you are going to make, but it’s hard to transform or not quite in the shape that would prove your hypothesis. As you go looking, you run into some data that points to a broken problem, which adjacent to your original hypothesis. At this point, the vividness of data that points to an adjacent problem will suck you in to making a set of decisions which will steer you away from the problem you originally intended to solve.
What’s worse is that after the product is launched and adopted, the data will show the problem is solved, and you’ll pat yourself on the back. All this while, what you did was solve a problem that data readily made visible, not the most important problem
This is one of the most important traps that product teams fall in to. That’s why it is very very important to articulate a hypothesis before looking for data to validate it, instead of digging through data to find all kinds of problems.
Game Changing products are a means of discovering the world
Like paradigm-defining science, game changing products often follow a path where the team working on the product notices that users are using the product in ways the team didn’t expect them to. This is followed by the team quickly digging a little bit deeper, and then iterating to expand on the functionality that their users loved. That is to say, the journey of a successful product is often filled with accidents.
The best way to increase these types of accidents is to cycle through product ideas quickly, and ensure that these ideas are in line with the original vision of what’s broken in this world.
Learn by Building 
“Poor decisions based on easily accessible data”… AMEN! I would also add to this list, manipulating easily accessible data to make it “say” what you want and support initiatives, when the data is actually not relevant to the outcome.