One Simple Logic Error That Can Undermine Your Growth Strategy
Survivorship bias is one of the easiest logical errors to make in Growth. If you’re not familiar with survivorship bias, it is best explained with a short example. During World War II, the US wanted to add a limited amount of armor to their long-range bombers in order to minimize their losses without compromising fuel efficiency or payload size. To do this, the military surveyed planes which had seen combat and decided to place the armor in the areas that were receiving the most damage. The military’s top brass was moving along with the plan until a statistician, Abraham Ward, came along and pointed out that this was actually the worst place to put the armor. He had the insight that all the surveyed planes had been able to withstand the damage in those areas and make it back to base. Abraham’s conclusion was that the armor should actually be placed in the areas where all of the surveyed planes received no damage, since the planes that were hit in those areas did not make it home.
In Growth, there are a few different ways that survivorship bias can manifest itself. In this post, I’ll cover three examples of how I’ve seen survivorship bias undermine people’s understanding of Growth and Growth strategies.
Why Conversion Rates Sometimes Decay
One of the ways survivorship bias can crop up is when conversion rates decay over time. Let’s say you launch a new signup prompt on a landing page or a new invite email that is particularly effective and significantly boosts your conversion rate. Immediately after shipping the experiment, everything looks great. However, when you come back 6 months later, you see that the conversion rate has steadily dropped back to your old baseline from before the experiment launch. What is happening here? Well, when you first launched the new signup prompt, it was something new that no one had seen before. Some percentage of people seeing the prompt for the first time decided to signup. Those people are now no longer part of the population seeing the prompt because they have accounts. After that, a portion of people seeing the prompt for the second time decides to signup. Again, they are now removed from the population. For companies growing at scale, eventually you’re left with a significant portion of the population who may have already seen that particular prompt 5 or 6 times and weren’t convinced during those first 5 times and will likely continue to not be convinced from seeing the exact same prompt. What this means for a Growth team is they need to budget time to revisit important conversion prompts with the expectation that as users get acclimated to the prompt, its effectiveness will start to decline. On a broader scale, it might also mean you need to change up your entire conversion strategy as you move further along the S-curve of adoption.
Existing Users Are Biased
Another way that survivorship bias can manifest itself is in your existing user base. Every active user you have today has figured out how to use the product and is getting enough value to continue to use it. Everyone else that didn’t get it has probably already churned out. When I was at Shopkick, the Growth team placed a big bet in building geofenced notifications to alert users to the number of points they could get when they were near one of our partner stores. The initial experiment results showed only a 2-3% lift in store visits, which was far below what we had been hoping for. We spent several weeks trying to debug the feature and trying to figure out why it wasn’t driving more visitation. After wasting several weeks, we were going to just chalk it up as a minor win and stop investing more in geofenced notifications. Just before we were about to give up however, we decided to segment the experiment analysis by new users vs. existing users and that is when we realized what was really going on. Amongst new users, we were seeing a 30% lift in store visits. In contrast, it was only about a 2% lift amongst existing users. This was surprising since we had not planned this as an activation experiment. Ultimately, this made a lot of sense. All our existing users had already figured out how to use the product and developed the habit of pulling it out when they were at the store. Everyone who hadn’t figured that out had already stopped using the app long ago. As a result of this insight, we decided to continue to invest more in geofenced notifications and grew it as a strategy for activating users.
The last example I’m going to talk about where survivorship bias can mislead you is when it comes to email engagement. A big area in email/push notifications is determining the optimal frequency to notify people. People often look at the email engagement of their existing users to try and figure out the optimal frequency to send. However, this can be a heavily biased approach because your existing users who are still receiving emails have self-selected themselves into being ok with your current frequency. Most people who disliked the frequency have already unsubscribed. To truly start to figure out the optimal frequency to notify users, you need to test on a clean unbiased population: new users that just signed up. Only by experimenting on new users, who have never received emails/notifications from your service before, can you determine what the optimal re-engagement frequency for your product is.
Recognizing some of the subtle ways survivorship bias can undermine what you think you know can allow you to adapt your growth strategy and help you decipher counter-intuitive results.