5 Principles For Goaling Your Growth Team

There are tons of posts on how to set a Growth team up for success from a hiring perspective, process perspective, tooling perspective, etc. However, one of the most important things to get right is setting up what each team is goaled on. This is especially true as a Growth Org starts to scale with multiple teams working in different areas, developing their own strategies, and each team figuring out how to drive their metrics up and to the right. It becomes critical that the goals and metrics are architected in a way to ensure each team is building towards a long term sustainable business.

Casey Winters wrote a great post on 5 mistakes to avoid when setting goals for a Growth team and why you should set goals on absolute numbers instead of percentages. In this post, I want to build on that and share 5 guiding principles I’ve found to be effective when architecting goals for a team, ensuring the team is aligned with the rest of the company and setting them up for long term success.

 

Principle #1: Isolation

The first key ingredient for a team level goal is making sure that it is a metric that the team can directly influence and has some degree of isolation from other teams, seasonality, etc. Team level goals should be a yardstick against which the team can measure progress. However, if that metric can be heavily influenced by other teams, by seasonality, or other factors outside of the team’s control, it starts to lose value as a way for the team to measure its impact and progress.

At Pinterest, the way each team in Growth measures its progress is by calculating the absolute number of incremental Weekly Active Users added by every A/B experiment run and summing up that impact over the quarter. By using A/B experiments to measure each team’s impact, the A/B experimentation framework provides isolation between teams, even when teams work in overlapping areas. The one downside of summing up experiment impact is that it has a tendency to overestimate the impact. Another approach is having team level holdouts for every experiment they run in the quarter, which can result in a more accurate measure of a team’s impact, but at the cost of significant additional engineering maintenance and overhead.

 

Principle #2: Avoid Misalignments

The second key ingredient of a team goal is making sure it does not put you at odds with other teams in the company. I recall on one occasion speaking to a Growth PM and they were complaining that their team goal was to grow the total number of registered users with the app installed. On the surface, this might seem like a fine metric. The problem was that this metric put them directly at odds with the Notifications team. This is because any time the Notifications team would send a push notification, some small portion of users will delete the app. Misalignments like this suggest that optimizing that metric might not correlate 100% to business success.

One possible way to overcome this is by having multiple teams in Growth all work to drive the same metric while defining swim lanes and areas of ownership to ensure each team has a clear charter. For instance, at Pinterest, most teams in Growth goal on Weekly Active Users, and each team has clear areas they own such as Emails & Notifications, or Signups & Logins.

 

 

Principle #3: Tied to Long Term Success

            The third key ingredient of a team metric is tying it to long-term business success. Growth teams are hyper metric focused and even with the best intentions, they can sometimes over optimize for a metric. To counteract this, you want to ensure that metrics cannot easily be “gamed” and that they are tied to the long-term success of the company. You can do this by going deeper than the surface level metrics and setting goals based on down funnel engagement.

A simple example is signups. It is natural for a team focused on user acquisition to perhaps goal on signups as the metric they drive. However, signups are a metric that can relatively easily be juiced in ways that might not lead to the best long-term outcomes. For instance, you can remove steps from the signup flow to get more users through the signup funnel, but those steps may be important in significantly improving the user experience once the users are in the product. Or perhaps, the acquisition team starts driving a ton of signups from low-quality traffic sources and signing up a ton of users that end up not sticking around. A better metric for a team focused on acquisition might be a metric like activated signups, where they only count signups that result in a long-term retained user. In this way, the team is ensuring the metric they are trying to drive is always aligned with long term business success.

 

Principle #4: Guardrails

Even with metrics that are tied to long term success, it can sometimes be necessary to setup guardrails to protect the user experience. A great example is emails and notifications; sending more emails always lifts engagement. Even if those additional emails result in more unsubscribes, those unsubscribes never overcome the lift in active users from just sending more. Intuitively though, everyone knows that there can be a long-term cost to sending too many emails which can cause user fatigue or brand perception. The problem is the long-term cost doesn’t appear in days or weeks, but over years.  To help protect the user experience, the team needs to establish guardrail metrics around unsubscribe rates and spam complaint rates to help protect the user experience while they drive towards their goal. The guardrail metrics would ensure that optimizing their northstar metric of active users isn’t at the expense of user experience.

Guardrails can also be helpful as a cross-check for the main metric the team focuses on to help catch potential issues. For instance, if the Emails & Notifications team decides to measure its impact through A/B experiments, it is important to setup a cross-check metric for the team to track the health of the channel outside of experiments. If a particular ISP starts marking the email as spam, it could significantly impact the business and the team needs to be able to catch that. The team would need to setup a guardrail metric like daily email clicks, or even better, daily engaged sessions from email, to ensure the team can catch and identify issues that fall outside of experiments.

 

Principle #5: KISS

            Finally, there is a well-known principle in Computer Science called KISS, which is an acronym that stands for “Keep it simple, stupid.” This principle was originally created by the Navy and states that most systems work best if they are kept simple rather than made complicated. With metrics, it can be easy to get a bit carried away and create extremely complex metrics or have several different metrics that a team is responsible for driving. When designing team metrics or a team goal, try not to overcomplicate things and just keep it simple, stupid.

The 27 Metrics in Pinterest’s Internal Growth Dashboard

One question I often get asked by people starting out on growth is “what metrics should be in my growth dashboard?”. I’ve written before about what metrics we value at Pinterest. In this post however, I’ll give people a peek behind the scenes and share what our internal growth dashboard looks like.

We have organized our dashboard to reflect our user growth model. We start with our top line growth metric of MAUs. Then we follow the user lifecycle funnel; starting with acquisition metrics, followed by activation, engagement, and finally resurrection.

Pinterest Growth Model

 

MAUs

1. Current progress to goal: Current number of MAUs & how much progress we’ve made towards our quarterly MAU goal.

2. MAU Forecast: Forecast of the number of MAUs we could expect to have extrapolated from our growth rate at the same time during previous years. We include this metric to help us anticipate the effect of seasonality on our growth numbers.

3. MAUs by app

4. MAUs by gender

5. MAUs by country: Tracking total number of MAUs in every single country would obviously be overwhelming to view on a chart, so instead we bucket countries together. The buckets we use are USA, Tier 1, Tier 2, Tier 3, and Rest of World. The tiers are based on size of Internet population, Internet ad spending, etc.

6. MAU Accounting: The MAU accounting helps us see what factors are contributing the most to our MAU growth. Specifically we split out total number of signups, resurrections, existing users churning out and new users churning out.

Growth MAU Accounting

 

Acquisition

7. Total signups

8. Signups by app

9. Signups by referrer

Signups by referrer

10, 11, 12. Invites Sent, Unique Invite Senders, and Invite Signups

 

Activation

 13. Overall Activation Rate: 1d7s is a term we use to refer to users who come back 1 or more times in the week following signup. We measure overall activation rate as 1d7s/signups, or in other words, the percentage of new signups that visit Pinterest again in the week following signup.

 14. Activation by app: This is the same metric of 1d7s/signups split out by platform. We’ve seen that different platforms can actually have pretty dramatic differences in activation rates.

15. Activation by referrer source

16. Activation by gender

Activation rate by gender

17. Overall signups to 1rc7: This metric is similar to the signups to 1d7 except it measures the percentage of new signups that repinned a pin or clicked on a pin in the week following signup. We use this metric to measure as a leading indicator of how well we are activating users into the highly engaged user buckets

 18. 1rc7s by app

19. Signups to 1rc7 ratio by app

20. Signups to engagement funnel by app: This metric tracks the percentage of new signups that are still doing key actions during a one-week time window of 28-35 days after signup. Specifically, we track 35 days after a user signs up, what percentage of them are still an MAU, WARC (weekly active repinner or clicker), WAC (weekly active clicker), or WAR (weekly active repinner).

21. Signup engagement funnel by gender

22. Signups to WAU 35 days after signup: This is one of our key activation metrics. We track the total percentage of users who are still a WAU one month after signup.  Specifically we look to see what percentage of signups were active between 28-35 days after signup.

 

Engagement

 23. *AU ratios: We track the ratio of DAUs to MAUs, WAUs to MAUs, and DAUs to WAUs. The ratio between *AUs is a popular metric to gauge how engaged users are with your app.

*AU ratios

24. Email Summary by type: Table of total number of emails sent, opened, & clicked-through split out by email type.

25. Push notification summary by type: Table of total number of push notifications sent & opened split out by type and by platform (iOS & Android).

 

Resurrections

26. Resurrections by platform: Total number of users that were dormant for 28+ days, but then came back to Pinterest, split out by which platform they came back on.

27. Resurrections by referrer

Retention rates by referrer

To wrap up, you can see we put a big emphasis on activation (the process of getting a new user to convert to a MAU). This is because we consider activation critical to long-term sustainable growth. Strong activation rates are necessary if you want to be able to scale a service to hundreds of millions of users.  We also put an emphasis on segmentation by gender, country, referrer, etc., to more deeply understand how different segments of users interact with Pinterest and see which segments are underperforming. If you have any questions feel free to ping me on twitter or drop me a line.

4 Metrics Every Growth Hacker Should Be Watching

The metrics typically advertised by startups are total users, daily active users (DAU), and monthly active users (MAU). While these numbers might be good to share with the press, they are only vanity metrics because they don’t give any real insight into your growth rate or the quality of the users you’re bringing in. Here are 4 metrics you should really be paying attention to if you’re trying to drive sustainable user growth.

Daily Net Change – Daily net change tells you on a daily basis how much you’re user base has grown (or shrunk). In a single graph you can assess new user acquisition, re-engagement, and retention and can easily see the impact of each component on your current growth rate.

Here is the breakdown of the different components and how they are calculated:

New Users: how many new users joined the service today?

Reactivated Users: how many existing users used the service today for the first time in 28 days?

Churned Users: how many existing users last used the service exactly 28 days ago?

Net change: new + reactivated – churned

Net Change Graph

Growth Metric: Net Change in Users

 

Core Daily Actives – The problem with the daily active user metric is there is not concept of quality users or retention. You will often see DAUs jump from a user acquisition campaign, but it is impossible to tell from the metric if those users are immediately dropping off or if they are sticking around. Core Daily Actives rises above this noise by only counting users that have been using your service on a regular basis. To get this metric, you calculate the number of users that used your service today who also used your service 5 or more times in the past 4 weeks. This metric is much more useful than DAUs because it focuses on the bottom line: growth of repeat users.

Cohort Activity Heatmap – The cohort activity heatmap is by far my favorite because it is the most insightful metric on this list. What the metric shows is how your user retention curve has changed over time. It can be a bit complex to read at first because so much data is crammed into a single graph, but it is very powerful once you know how to use it.

This is how you interpret the graph:

  • The unit of the x-axis is days and each column corresponds to the group of users that joined on day X (each group is called a cohort)
  • The width of a column on the x-axis represents the size of the cohort (i.e. the wider the column, the more users joined on that day)
  • The unit of the y-axis is also days and each row represents Y days after the cohort joined the service. The bottom row of the graph represents day 0, the very first day the user joined the service, and the top row represents day 59.
  •  The color of each rectangle represents activity level. It is calculated by determining the percentage of users that joined day X and used the service on day Y. The scale ranges from red for a high percentage to blue for a low percentage.

 Cohort Activity HeatmapGrowth Hacker Metric: Cohort Activity Heatmap

 

Conversion Funnel – The final metric is tracking the conversion funnel for flows that affect new user acquisition, retention or re-engagement. A conversion funnel is simply splitting up a process into its constituent steps and tracking how many users make it through each step. This metric is widely used, but it is common to analyze the conversion funnel for a flow once or very infrequently. Really, the conversion funnel for important flows should be tracked on a daily basis for adverse changes because even a difference of a few percentage points can compound over time.

This is an example of the conversion funnel for inviting users to a service.

1)   How many users saw the invite prompt?

2)   What percent of users clicked on the invite prompt?

3)   How many invites were sent per user that clicked on the prompt?

4)   What percent of invites were viewed?

5)   What percent of invitees clicked on the link in the invite?

6)   What percent of invitees that clicked on the link, joined the service?

7)   How many new users joined the service as a result of the invite.

 

This is just a sampling of some of my favorite growth hacking metrics. There are many others and usually the best metrics for you depends on your situation. If you know any metrics that you think should have been on this list, please drop me a line.

The Secret to Effective Geofencing

You’ve probably heard of geofencing, but in case you haven’t, geofenced notifications are alerts that users receive on their smartphone when they are near a particular location. They are a powerful activation & re-engagement tool for location based apps because they communicate information to a user at a time that is contextually relevant. Over the past year, geofenced notifications have become popular with many shopping apps such as RetailMeNot, RedLaser, etc. In this post I’ll cover some of the basic problems you might encounter with developing geofencing.

1. Inaccurate Location Data
The dirty little secret of most location data providers is that their data just isn’t that accurate. Typically for geofencing you will want to geofence a small radius (ex: 150 meters) around the point of interest. The way these data providers get latitude & longitude information is through a method called address interpolation. The problem is that the latitudes and longitudes from this method can often be off by hundreds of meters. This means when a user is standing directly at the point of interest, there is a good chance they will not fall into the geofencing radius! To overcome this problem you either have to manually fix the location data (like eBay did) or use user data to automatically correct the locations.

2. Battery Drain on Android
Android has built in support for geofencing, but unfortunately it has severe battery drain issues on certain versions & devices. To work around this issue, many developers have taken to developing their own proprietary geofencing framework based on polling GPS. However, a lot of attention needs to be given into tweaking the framework to ensure users receive notifications in a timely manner, but without significantly draining their battery. Some ways to decrease battery drain are to poll less frequently, or instruct the device to only use wifi data to get a location fix rather than turning on the GPS.

3. iOS’s hard limit on the number of geofenced regions
Thankfully, iOS has a geofencing framework that doesn’t seem to drain user’s batteries. Unfortunately, iOS sets a pretty low limit on the number of regions you can monitor. This means to geofence effectively, you need to update which regions your geofence as a user moves around. Luckily, iOS has another framework that will wake up your app when the user has moved a significant distance (on the order of 5-10 miles). You can use this framework to wake up and update the regions your app is geofencing.

These are just a few of the problems that might be encountered when developing geofencing, but geofencing is such a powerful engagement tool and it is well worth the effort.

3 Lawsuits Every Growth Team Needs To Know

I’ve found that people in Growth are all too often not aware of some of the major laws that govern the industry. In this post, I’ll cover three major lawsuits that sued over growth tactics used by well-known companies and some principles for how your Growth team can stay out of trouble.

 

Hickey v. Voxernet LLC

In 2012, Voxer was sued over their invite flow where users sent SMS invites to their friends. Instead of just popping up the SMS app with a pre-populated message, the flow in Voxer was it would popup a contact list picker and have the user select a number of different friends from their contact list to invite. After the user selected the friends they wanted to invite then Voxer sent those friends an invite from their servers. The claim was that Voxer sending SMS invites violated a clause in the Telephone Consumer Protection Act (TCPA) against Automated Telephone Dialing Systems (known as ATDS or auto-dialers). Voxer ultimately settled, and in the three years following that lawsuit, a number of lawsuits were filed against other companies that also had similar flows which used a contact list picker to allow users to send multiple SMS invites [Sterk v. Path Inc.][Huricks et al v. Shopkick, Inc][McKenna v. WhisperText LLC][Glauser v. GroupMe, Inc.][Reichman v. Poshmark, Inc]. A lot of the lawsuits stemmed from some ambiguity on what was considered an auto-dialer. Then in 2015, the FCC issued a ruling to clarify the ambiguity and laid out clear guidelines for when SMS invites were ok. Namely, they needed to meet the following criteria:

  • The invites required human intervention (i.e. they had to be initiated by the user) The user needs to play a role affirmatively deciding to send a text message, selecting who to send to, and preferably controlling the contents of the text message.
  • It had to be clear to users that their friends were going to receive a text message and how many messages would be sent as a result of their action (as opposed to a different communication channel such as an email).

The biggest takeaway is that while sending SMS invites can be very powerful, they can also open a company up to a lot of legal risk. To mitigate that risk, be very thoughtful about how you design your invite flow and preferably get a lawyer to carefully review your invite flow and provide a written legal opinion with screenshots of the flow.

Poshmark SMS invite

Perkins v. LinkedIn Corp.

LinkedIn was sued in 2013 for sending invite reminder emails. When a LinkedIn user invited their friends or colleagues to the service, LinkedIn would send an email to those people. If they didn’t respond to the initial email, they would receive a 2nd and a 3rd email as a follow up reminder to join the service. While on the surface, that doesn’t sound very nefarious, the lawsuit alleged that users had consented to the 1st email, but they had not consented to the 2nd or 3rd reminder emails. The lawsuit also alleged that by sending reminder emails the user had not consented to, it could potentially damage the user’s reputation by spamming their contacts without their consent. It also alleged that users had not consented to their name and profile picture in the reminder email. LinkedIn ultimately settled for $13M, changed its product policy and privacy policy. The takeaway from this lawsuit is that even for email invites it needs to be very clear to users how many people will be contacted and how many times their friends will be contacted.

LinkedIn Invite Reminder Email

 

Opperman et al v. Kong Technologies, Inc. et al

In 2011 and 2012, many apps such as Instagram, Twitter, Yelp, Path, Kik, Gowalla, Foodspotting, and Foursquare were caught uploading users’ contact lists to their servers without user knowledge or consent. In most cases, the reason these companies were doing this was to do friend recommendations (to either identify friends already on the service or to recommend friends to invite). Apple quickly added the Contact List Permission to iOS 6 as a result of the scandal and the defendants agreed to settle the class action lawsuit for $5.3M. The takeaway from this lawsuit is that even if something is technically possible, and even if you are allowed by your products Terms of Service, you should be very clear and upfront when collecting data that a user would not reasonably expect be collected.

iOS 6 contact list permission

Conclusion

Path Inc. in particular got hit with a triple whammy. Not only was it sued for SMS invites in Sterk v. Path and had to pay into the Opperman et al v. Kong Technologies, Inc. et al settlement, it was also fined $800,000 by the FCC for violating the Child Online Privacy Protection Act (COPPA). The violation stemmed from the fact that a few thousand children under age 13 were allowed to sign up for the product and Path collected personal information from those users without parental consent.

Growth teams and startups can have a mentality of “move fast and break things.” However, it is important to understand what areas to pay careful attention to if you want to avoid the company getting bogged down in lawsuits and legal wrangling: SMS, invites, referrals, and data privacy. These two principles can help you stay out of trouble:

  • Make sure the Growth team is aware of major laws that impact their work like the Telephone Consumer Protection Act (TCPA), Child Online Privacy Protection Act (COPPA), European General Data Protection Regulations (GDPR), etc.
  • Make sure you always get informed user consent any time you are collecting data the user might not expect, or when the user is going to send an invite/referral.

 

Disclaimer: I am not a lawyer and this post in no way constitutes legal advice

Growth Engineering Meetup (Airbnb, Pinterest, Twitter)

Pinterest recently hosted a SF Growth Engineering meetup at the Pinterest office to discuss how some of the most successful companies drive growth at scale. See the video below to hear what speakers from Pinterest, Airbnb and Twitter had to share.

Synopsis of the talks

Paid growth @ Airbnb

Ganesh Venkataraman discusses the various channels and techniques used towards paid marketing at Airbnb as well as the underlying challenges around data/machine learning and attribution.

Speaker: Venkataraman leads guest growth engineering at Airbnb. Previously he worked at LinkedIn where he led careers relevance (job search, recommendations and statistical insights on salary). He has over 20 publications/tutorials/talks in leading conferences including SIGKDD, SIGIR, CIKM, SODA etc. He also has filed 16 patents in areas related to search and machine learning.

Onboarding component framework @ Twitter

Michael Lin covers Twitter’s Onboarding Component Framework which allows backend systems to construct personalizable signup flows using generic client-side components. This talk addresses the infrastructure, experiments and learnings from building this system.

Speaker: Lin is an infrastructure engineer on the Onboarding Team at Twitter. During his four year tenure at Twitter, he has also worked on account security, anti-spam and ads infrastructure. Michael is a graduate of the University of Toronto.

Keeping your resurrected users out of the graveyard @ Pinterest

Pinterest has millions of dormant users that resurrect each week, but often times they churn back out. Allan Blair discusses how Pinterest has built a strategy to classify, measure impact and ultimately increase our resurrected user retention.

Speaker: Blair is a full-stack engineer on the Engagement Growth team at Pinterest. Previously he worked as an Investment Banker at J.P. Morgan on Wall Street. Allan is a graduate of Rensselaer Polytechnic Institute in upstate New York.


The SF Growth Engineering Meetup holds these talks like these about once a quarter. Feel free to j. Join the SF Growth Engineering Meetup Group to stay in the loop and find out about future events.

AMP In Email And What It Means For The Future of Email Marketing

It is not often that something new comes along in the email space, much less something that has the potential to change email marketing as we know it. In case you missed it this week,  Google announced plans for Gmail to support AMP (Accelerated Mobile Pages) across all Gmail clients. While it isn’t immediately available today, Google expects it to be available sometime later this year. For CMOs, Heads of Growth, or anyone interested in Growth or Email Marketing, here is a primer on what AMP in email could mean for you.

What is AMP in email?

For a couple years now, Google has been promoting AMP as a standard that websites can use to make fast and responsive mobile web pages. AMP is a variant of HTML that has a number of restrictions on things that can sometimes make websites slow. AMP also adds a number of new HTML elements for common interactive UI components.

For those who have been lucky enough to avoid ever having to write email HTML, I’ve heard it best described as writing HTML as if it was 1999. There are a huge number of different combinations of email clients, operating systems, email service providers, and browsers. There are a lot of quirks or bugs in different email rendering engines, which means you have to write HTML for the lowest common denominator. Slowly over the past decade, some of the worst clients have been aging out and there has been an increasing trend towards something called “Kinetic Email”. Kinetic Email, which has been fully supported on web for over a decade, takes advantage of some creative hacks of GIFs combined with some basic CSS animations to make emails a little bit more interactive.

AMP in email takes the trend of Kinetic Email and blows it out of the water by now allowing you to write fully functional (albeit simple) web applications in email. You can literally now deliver a simple interactive web application straight into people’s inboxes. To see an example, watch this video of what Pinterest was able to do with AMP in email.

 

What does this mean for Growth?

Growth people will need to be thoughtful with how they use this. Growth people often want to bring users back to the product since that is where you can make revenue (ad impressions, purchases, etc.) There are a few strategies you will need to think through for AMP in email:

1) Do you use AMP in email to improve the chances of someone coming back in the future? For Pinterest, activation is often about gathering signals from users, so getting them to give us signals about their interests can improve retention even if they stay within Gmail and don’t click through to the product.

2) Do you use AMP in email simply as a tool to boost your overall CTR rate by making the email more interactive and engaging?

3) If users can now engage in core features of the product in email (i.e. make bookings, purchases, etc.), do you start to think of email as another client in addition to Web, iOS, and Android? Do you start to count people who take actions through your AMP email as DAUs?

 

What are the challenges ahead?

1)  AMP in email is an open specification, but we will have to see if this remains a Gmail specific feature or if other industry players (ESPs, marketing tools, WSIWYG design tools, etc) begin to adopt it. If it does remain a Gmail only feature, Gmail has such a large market share that it will still be worth adopting for any company that relies on email for re-engagement. Development time is on-par with traditional email development, even though the functionality is much richer;  it shouldn’t need to be a huge investment for companies.

2) Security. Gmail has already put a lot of thought into security with regards to phishing attacks, etc. However, as companies start to adopt this and explore what is possible, you can be sure scammers will be quick to exploit any potential vulnerabilities in user behavior.

3) Consumer adoption. This opens up a whole new way of interacting with emails. While it may be easy for companies to experiment with interactive email, there will be a learning curve for consumers to really understand that they can now engage with emails beyond just clicking a CTA button. The impact that AMP in email can have will increase as more and more consumers start to comprehend that they can interact more deeply with email and start to become more comfortable with doing more complex interactions over time.

 

Conclusion

AMP in email is exciting and perhaps one of the biggest potential innovations in email in the past decade. TechCrunch took a surprisingly luddite view on why “AMP in Email is a Terrible Idea“. Like with anything new, there are challenges ahead, but this has the potential to make email a lot more functional and engaging communication channel. To get started with checking out what is possible with AMP in Email register for the developer preview.

 

Acknowledgements: Seth Weisfeld for co-authoring this post. Susan Su from Reforge for inspiring me to write this post.

All the Advice You’ve Read on Push Permission Prompts is Wrong

For the last few years, it has become a widely promoted industry best practice to have pre-prompts for iOS’s push notification permission. This advice first originated from this 2014 blog post by a small startup called Cluster, which got picked up by TechCrunch at the time. Since then, it seems like every mobile analytics company has written a blog post repeating this advice: Localytics, Braze (formerly AppBoy), Optimizely, CleverTap, UrbanAirship, and the list goes on. The only problem is that, at least when it comes to push notification permissions, this advice is flat out wrong.

For those who aren’t familiar with the problem, you only get one shot when you pop up the system permission dialog for the push notification permission on iOS. If the user declines to give you the push permission, you cannot send them a re-prompt ever again. The theory behind pre-prompts is that you use pre-prompts to hold off showing the system permission dialog until the user is primed to accept. While this all makes complete sense in theory, it falls apart in practice.

For the past 4 years, I’ve worked on the Emails and Notifications team at Pinterest. During that time, we’ve tried more than a half-dozen different attempts at using pre-prompts to improve our opt-ins on push notifications. We’ve not only experimented on iOS but have also experimented on the permission prompts for browser notifications as well (Chrome, Firefox, Safari, etc.) The attempts have ranged from very simple pre-prompts with multiple re-prompts to complex flows to try and prompt users at a time where they could understand contextually why they should turn on push notifications. At my previous company, Shopkick, we even tried more “growth hacky” type approaches like putting a pulsar underneath the “allow” button to not so subtly suggest which button they should click. Even though all these experiments took different approaches, different framing, and different designs, none of them were successful. Every single one of these experiments showed a drop in engagement metrics across the board when compared to just popping up the system prompt.

In spite of all these experiments, I thought maybe we were just getting something wrong. Maybe there was something we were missing? I went and talked to other people working on push notifications at scale to find out. I’ve met with people working on notifications at a social network company based in Menlo Park, a music streaming company, a professional social network, and many more. When I talked to people at these companies working on notifications at scale, they all told me the same thing. They had all tried pre-prompts (and often had tried several iterations), but they had never found anything that beat just popping up the system permission dialog.

Pre-prompts aren’t all bad. I’ve seen them work for other permissions like photos or microphone. However, when it comes to push notifications, users make up their mind based on the category your app is in and there is little you can do to sway it. So, don’t put users through the hassle of two prompts just to turn on notifications, or even worse, put them through a pre-prompt and multiple re-prompts when they don’t want notifications. Just pop up the system dialog and get it over with.

P.S. If you’ve had success at scale with pre-prompts for the push notification permission, reach out to me at [email protected]. I’d love to hear about your experience.

3 Steps to Cracking International Growth

Cracking international growth is critical for any product that aspires to grow to hundreds of millions or billions of users. One of the key challenges with growing internationally is you can’t rely on your own intuition and experiences the same way you can when trying to grow in your native country. When I joined Pinterest, it was predominantly a US based service and had only just begun the process of starting to grow internationally. Over the next few years however, the company figured out what it took to grow the product internationally and mid last year we crossed a major milestone when over 50% of our active users were based outside of the US. In this post, I’ll cover the three fundamental problems a service needs to solve to be successful growing internationally and how to solve them.

The first and most basic problem a company needs to solve is localizing the product. At the most basic level, this means setting up a system to translate every line of text in the product. What people new to growth sometimes fail to appreciate is how much work it takes to really do this well. The words used in the product are the words new users read to understand the value of the product and how to use it. If the descriptions in the product don’t make a lot of sense, users will have a hard time figuring out how to use it and will ultimately be more likely to churn. At Pinterest, we saw a powerful example of how significant of an effect localization can have on comprehension and growth when we changed the text on the “Pin it” button to instead say “Save.” In user research, we had found that the metaphor of pinning things to corkboards wasn’t as prevalent in some cultures and therefore the entire metaphor of Pinterest as a product didn’t make as much sense. After changing “Pin it” to “Save,” we saw significant increases in Growth metrics in non-English speaking countries. As a result, we’ve started to take a very metrics-oriented approach to localization by building Copytune, a framework that allows us to test copy and pick the best performing variant on a per language basis.

The second problem is getting product/market fit in international markets. Just like any new startup needs to reach product/market fit when are first starting out, you need to revisit if you have product/market fit in important international markets when you begin to expand globally. One great example of this was when Uber added a cash payment option in India because many Indians did not have bank accounts or credit cards. Another example was when Facebook added blood type as a profile field in Japan because the Japanese believe that blood type imbues a lot of traits on a person’s character and is an important part of modern Japanese culture. It could even be as simple as improving performance so the site doesn’t take 20 seconds to load. For Pinterest, we ran several user research studies in key markets when we began expanding internationally. We discovered that while the website was properly localized when an international user signed up, their homefeed would be full of American pins with English descriptions. For us, reaching product/market fit was about updating our algorithms that powered search, homefeed, etc. to ensure that we surfaced locally relevant content. In other words, when a French user searched for recipes, they should find suggestions for local French cuisine unless they explicitly tried searching for international cuisines.

The final problem is building out your playbook for how to unlock Growth in a new country. After a startup first starts to gain traction, they go on a discovery process to figure out which growth channels work for them (SEO? Emails? Notifications? Paid Ads?, etc). Nailing international growth requires figuring out how to scale your tried and true growth channels to international markets. A great example of this again comes from when Facebook expanded into Japan. Facebook relied heavily on viral invites, but Japanese users initially didn’t send many invites. After doing some user research, they discovered that Japanese people were much less likely to send an invite to their friends asking them to join the service because they were worried their friends would find it bothersome or rude. However, when Facebook changed the language from “invite your friends” to “announce you joined Facebook,” they saw a big uptick in invites. Japanese users did not perceive announcing their presence on the service to be rude in the same way they perceived asking a friend to join the service. Other companies have had success in Asia when they adapted their invite flow and added locally popular messaging platforms such as Line, WeChat, Kakao Talk, etc. as options in their flow. You ultimately want to build up a playbook for how to tackle analyzing and building up your growth channels that you can then repeat on a country by country basis as the company starts to expand.

International Growth is a tough problem, and I’m only able to scratch the surface in this post. When I joined Pinterest, I was eager to learn how to scale a company internationally since my previous Growth work at Shopkick had only been based in the US. What I’ve learned is that success growing internationally requires taking many of the lessons from scaling in your native market and then figuring out how to adapt them to different countries and cultures. It requires nailing comprehension, making sure you have product/market fit for each country, and figuring out how to adapt your growth channels for individual markets.

One Simple Logic Error That Can Undermine Your Growth Strategy

Survivorship bias is one of the easiest logical errors to make in Growth. If you’re not familiar with survivorship bias, it is best explained with a short example. During World War II, the US wanted to add a limited amount of armor to their long-range bombers in order to minimize their losses without compromising fuel efficiency or payload size. To do this, the military surveyed planes which had seen combat and decided to place the armor in the areas that were receiving the most damage. The military’s top brass was moving along with the plan until a statistician, Abraham Ward, came along and pointed out that this was actually the worst place to put the armor. He had the insight that all the surveyed planes had been able to withstand the damage in those areas and make it back to base. Abraham’s conclusion was that the armor should actually be placed in the areas where all of the surveyed planes received no damage, since the planes that were hit in those areas did not make it home.

In Growth, there are a few different ways that survivorship bias can manifest itself. In this post, I’ll cover three examples of how I’ve seen survivorship bias undermine people’s understanding of Growth and Growth strategies.

Why Conversion Rates Sometimes Decay

One of the ways survivorship bias can crop up is when conversion rates decay over time. Let’s say you launch a new signup prompt on a landing page or a new invite email that is particularly effective and significantly boosts your conversion rate. Immediately after shipping the experiment, everything looks great. However, when you come back 6 months later, you see that the conversion rate has steadily dropped back to your old baseline from before the experiment launch. What is happening here? Well, when you first launched the new signup prompt, it was something new that no one had seen before. Some percentage of people seeing the prompt for the first time decided to signup. Those people are now no longer part of the population seeing the prompt because they have accounts. After that, a portion of people seeing the prompt for the second time decides to signup. Again, they are now removed from the population. For companies growing at scale, eventually you’re left with a significant portion of the population who may have already seen that particular prompt 5 or 6 times and weren’t convinced during those first 5 times and will likely continue to not be convinced from seeing the exact same prompt. What this means for a Growth team is they need to budget time to revisit important conversion prompts with the expectation that as users get acclimated to the prompt, its effectiveness will start to decline. On a broader scale, it might also mean you need to change up your entire conversion strategy as you move further along the S-curve of adoption.

 

Existing Users Are Biased

Another way that survivorship bias can manifest itself is in your existing user base. Every active user you have today has figured out how to use the product and is getting enough value to continue to use it. Everyone else that didn’t get it has probably already churned out. When I was at Shopkick, the Growth team placed a big bet in building geofenced notifications to alert users to the number of points they could get when they were near one of our partner stores. The initial experiment results showed only a 2-3% lift in store visits, which was far below what we had been hoping for. We spent several weeks trying to debug the feature and trying to figure out why it wasn’t driving more visitation. After wasting several weeks, we were going to just chalk it up as a minor win and stop investing more in geofenced notifications. Just before we were about to give up however, we decided to segment the experiment analysis by new users vs. existing users and that is when we realized what was really going on. Amongst new users, we were seeing a 30% lift in store visits. In contrast, it was only about a 2% lift amongst existing users. This was surprising since we had not planned this as an activation experiment. Ultimately, this made a lot of sense. All our existing users had already figured out how to use the product and developed the habit of pulling it out when they were at the store. Everyone who hadn’t figured that out had already stopped using the app long ago. As a result of this insight, we decided to continue to invest more in geofenced notifications and grew it as a strategy for activating users.

 

Email Unsubscribes 

The last example I’m going to talk about where survivorship bias can mislead you is when it comes to email engagement. A big area in email/push notifications is determining the optimal frequency to notify people. People often look at the email engagement of their existing users to try and figure out the optimal frequency to send. However, this can be a heavily biased approach because your existing users who are still receiving emails have self-selected themselves into being ok with your current frequency. Most people who disliked the frequency have already unsubscribed. To truly start to figure out the optimal frequency to notify users, you need to test on a clean unbiased population: new users that just signed up. Only by experimenting on new users, who have never received emails/notifications from your service before, can you determine what the optimal re-engagement frequency for your product is.

 

 

Recognizing some of the subtle ways survivorship bias can undermine what you think you know can allow you to adapt your growth strategy and help you decipher counter-intuitive results.