John Egan

When it comes to growth, one potential pitfall is over optimizing for short-term wins. Growth teams operate at a pretty fast pace, and our team is no exception. We’re always running dozens of experiments at any given time, and once we find something that works, we ship it and move on to the next experiment. However, sometimes it’s important to take a step back and validate that a new tweak or feature really delivers long-term sustainable growth and isn’t just a short-term win that users will get tired of after prolonged exposure. In this post I’ll cover how we optimize for long-term sustainable growth.

Last year, we started including a badge number with all our push notifications. For many people, when an app has a badge number on it, the impulse to open and clear it is irresistible.

Figure 1: Example of badging on the Pinterest iOS app

When we launched badges, we ran an A/B experiment, as we do with any change, and the initial results were fantastic. Badging showed a 7 percent lift in daily active users (DAUs) and a significant lift in other key engagement metrics such as Pin close-ups, repins and Pin click-throughs. With such fantastic results, we quickly shipped the experiment. However, we had a nagging question about the long-term effectiveness of badging. Is badging effective long-term, or does user fatigue eventually set in and make users immune to it?

 To answer this question we created a 1 percent holdout group. A holdout group is an A/B experiment where you ship a feature to 99 percent of Pinners (users) and keep 1 percent  from seeing the feature in order to measure the long-term impact. We will typically run a holdout group whenever we have questions about the long-term impact or effectiveness of a particular feature.

We ran a holdout experiment for a little over one year. What we found was that the initial lift of 7 percent in DAUs settled on a long-term baseline of a 2.5 percent lift in DAUs after a couple months (see mobile views in Figure 2). Then last fall we launched a new feature, Pinterest News, a digest of recent activity of the Pinners you follow. As part of News, we would also badge Pinners when there were new News items. As a result, News helped increase the long-term lift of badging from 2.5 percent to 4 percent.

Figure 2: Lift of the badging group over the holdout group for key engagement metrics

We also found that badging was effective at increasing engagement levels. We classify Pinners into core, casual, marginal, etc., and we found badging had a statistically significant impact on attracting those who would have fallen in the marginal or dormant bucket to instead become core or casual users. This finding was compelling since it proved that badging is effective at improving long-term retention.

Figure 3: The badging group drove more Pinners into higher engagement buckets when compared to the holdout group

Holdout groups have been an effective way for us to ensure we’re building for long-term growth. We also have hold out groups for features like ads, user education, etc. In general, holdout groups should be used anytime there is a question about the long-term impact of a feature. In the case of badging, it allowed us to understand how Pinners responded to badging over a prolonged period of time, which will help inform our notification strategy going forward.

This article was also published to the Pinterest Engineering Blog