5 Common Myths of Data-Driven Decision-Making That You Probably Didn’t Know
I love data. And over the course of my time learning about data and working with data-driven strategies, I’ve come across a lot of people who are living by this data-driven approach as well. As much as I think this is great, there are a few things that I feel should be addressed.
The term “data-driven” has a strong power that makes people believe they can achieve anything and change the world! What isn’t talked about is how often data can be presented out of context or lead to some bad decisions if not analyzed appropriately.
To help you better understand it, here are the top five myths and misconceptions using data to make decisions.
- Myth 1: Data-Driven Decision Making is Always Right
- Myth 2: Failing Performance Means Poor Data
- Myth 3: Better Results Means Your Data-Driven Strategy Is Accurate
- Myth 4: Data Will Never Be Biased
- Myth 5: You Can Acquire All of The Data You Need to Make a Decision
Contents
Myth 1: Data-Driven Decision Making is Always Right
While a data-driven approach makes sense, there are some circumstances where you shouldn’t let the data make the ultimate decision. One of the most common situations would be during the startup phase, when you haven’t accumulated enough data to inform a proper decision.
If you are building a startup or starting a new project/service, chances are you don’t have enough powerful data right from the jump. You might find yourself seeking out any small bit of data to help you in deciding on a vertical, or avenue, to focus on for the long term.
Rather than taking any small piece of data when you start out, don’t focus on making a data-based decision at that point. I would recommend spending more time testing your tests, and trying out several different tactics before you commit to a specific direction or vertical. I firmly believe that making as many moves as possible is equally (or MORE) important while you’re in the building phase.
Data has been around for eons, and the approach isn’t going anywhere soon. You can take advantage of lessons learned in the past and apply them to your future. The more data you cultivate, the more on point your strategies are likely to be.
For this reason, at some point you might need to focus doing more to take action rather than the data itself.
Myth 2: Failing Performance Means Poor Data
Eric Ries, author of Lean Startup, says we should celebrate the failure as a good opportunity to learn. A failure is a stepping stone to learn more about your business or even your product. Data works the same way in that it’s a continual process to improve. Unfortunately, this philosophy isn’t as widely embraced as it could be. I’ve witnessed situations where managers blow up after receiving low performances from their team after the first report (luckily not at me).
Extracting decisions from your data is a never ending process, and needs to be given the time to garner the best results. For example, I’ve been running a PPC campaign as part of the internal lead generation project, and the landing page that I used had gone through a major design iteration. When the new version of the design was tested, the results fared far worse than the original design when it came to generating conversions. That being said, it ended up generating almost double the conversions of the original (from 7 monthly conversions to 19 conversions) after a year. If we reverted back to the original or abandoned the strategy too soon, we would not have been able to increase our leads as well as we did.
The outcome of companies that usually reprimand their employees after the first round of collecting data is stagnancy. The likelihood is that people revert back to what was “working”. From there, the quiet downfall begins: they’re never going to test out something new or take an innovative approach to new challenges. They’re going to wind up stuck in a ditch of their own making, and before they know it, they’re dead in an ever evolving market and torn apart by competitors. When it comes to business, you have to evolve, change, and keep moving. If you keep making moves, you will eventually reveal valuable data that will open the doors for success (promise).
I’ve found that a lot of people don’t like change, they like ‘positive’ change, but naturally, change can be a tough pill to swallow. Take the pill of change — it’s a wild ride.
Myth 3: Better Results Means Your Data-Driven Strategy Is Accurate
This is the flip side of my previous point.
When you get better results, don’t go thinking those are the accolades of your data-driven approach. No one likes to doubt their own success, but at this point, you should fearlessly question everything — including positive results.
I’ve often encountered online articles claiming, “Hey, I’ve improved the conversion rate by 1% fixing ____ by doing ____. I’ve got better search ranking by ______.”
Okay.
How can you be entirely sure that the improvement is a result of what you actually did? You might say it’s the only thing you changed, so how could it be anything other than your genius? Sorry, Mr. or Mrs. Smarty-pants, your improvements, even though they’re driven by data, are not the only things that have changed during that testing period. Simultaneously, these external aspects have also been altered:
- Macro Trends
- Seasonality
- Competition
Most of the time, #2 and #3 are constantly in flux.
An area where this is a common problem is through data-driven case studies. It’s important to always think critically about published results until all external factors have been considered. I’m sure you’ve all read blog posts claiming to have improved performance by such and such %, simply by doing blah blah blah, yatta yatta, etc. However, I’m not sure about these claims. There needs to be trusted details, background information, and more context in order to understand the ‘blah blah blah, yatta yatta, etc’. Don’t believe the case studies — you need to have all the details to see exactly where these big results came from. You have to practice their method on your own to see for yourself (check out my 5 marketing roadblocks post).
To measure performance properly, you have to be laser focused to pursue the real contributors to success. Otherwise, you and your team won’t be able to see what is truly valuable from your results, and you risk making decisions based on misinformation. You definitely don’t want that.
Myth 4: Data Will Never Be Biased
Data is objective. Yep, I can’t argue that — it’s absolutely true. But, who is going to convert the data into new strategies or actions for you? If you use automated optimization for your ads, who is going to set it up? Who is analyzing this objective data? It’s us! We humble humans complement the data with our own reasoning in order to take action.
People love taking credit for success, regardless of whether they actually made a real contribution or not. In general, we humans love to make ourselves appear successful, and convince people to buy things. In a sense, we’re showing our bias.
Let’s say you have to report your website’s performance to your boss, who is a serious hard-ass (for the record, that’s not my boss), and you’re scared of your boss on a daily basis. In addition, you have no idea why the performance has gone down. The results are shown here:
| Week | Sessions | Session Growth | Conversions | Conversion Rate | Conversion Growth |
| Week 1 | 600 | 102% | 8.4 | 1.4% | 102% |
| Week 2 | 1100 | 83% | 20.9 | 1.9% | 149% |
| Week 3 | 1300 | 18% | 6.5 | 0.5% | -69% |
Now, you have to think about how the hell you are going to present this. Your choice might be 1) focusing mainly on the actual lower performance and figure a way out with your boss; 2) focusing mainly on the good parts — the small session growth, even though it might be just the margin of error. Luckily or unluckily, your boss is not familiar with those metrics. What would you do?
[Note: While this number might be a bit of a stretch, in the real world the situation and the metrics aren’t as simple as this table, and there are a lot of metrics that are not presented for good reasons and bad reasons.]
Another example might be that some case studies show their stats are achieving 60% session growth compared to last month. But when you take a look at the actual number, it turns out that it’s only grown from 100 to 160 sessions (again, this number can be just the margin of error.)
Depending on the way you present and see the data, the data becomes totally different.
Myth 5: You Can Acquire All of The Data You Need to Make a Decision
The data we cultivate is limited. You can never have the data you’ll feel 100% confident in to inform your decision making. Chances are, you’ll only ever be 50 – 80% sure.
Let me tell you a story to help explain what I mean. If your customer follows this journey…
One of your customers was browsing a few of their favourite online magazines which she keeps tabs on through her smartphone. She found out about your product, and now she’s interested in learning more. She, however, closed the browser because it was 2am — and needed her beauty sleep.
Two days later, she suddenly remembered your product in the middle of the day at the office, but there is one thing she forgot — she couldn’t for the life of her remember the name of your product.
A week later, she’s having a casual chat with her co-worker at the office. She once again encounters your product on her co-workers social media feed (albeit, it’s kind of rude that this person was checking their feed while chatting, but that’s a lesson for another day). She has found your product again — and now she’s buying it online.
It might sound crazy, but since we’re so often bombarded with products, services, news, and memes, it’s easy to forget about your product. In this scenario, you can’t keep track of what contributed to the conversion from where you’re sitting. And I’m pretty sure the customer doesn’t remember much about the journey either.
The lesson here: don’t rely too much on data alone and what you can see from where you’re sitting. Sometimes, simply increasing exposure works better than just focusing on cost-per-acquisition and click-per-cost, depending on what products you’re selling.
Wrap it up!
Being driven by data is better than being driven by guts alone, no question. But having said that, you need to know how to take advantage of the data in order to make your business a real success.
A lot of articles will use deceptive data in order to make their opinion more legitimate. Data, as a whole, has granted us a huge opportunity to doubt everything we thought might or might not work, and the ability to reveal the truth. Don’t let people crowd you with data. Data is always being biased towards whomever is handling it.
So doubt everything, if you want to make the best decisions. Doubt data, question everything, and explore unexpected angles. When you do that, you’ll always be led to the best decision.
If you liked this article, please check out my “The 3 Worst Mistakes That Devastated My Marketing Projects” as well.
If you want to learn more about specific key elements of an effective marketing plan, read our 50+ page booklet on How to Write A Strategic Marketing Plan That Yields ROI – complete with explanations
