How Pinterest Supercharged its Growth Team With Experiment Idea Review

Written by Jeff Chang & John Egan

Growth teams need to be organized bottom-up to scale well

The Pinterest Growth team has over 100 members, and we’ve run thousands of experiments over the years. It’s difficult to run that many experiments and still maintain a high success rate over time. We’ve found the traditional growth team model of team leads deciding which opportunities to try didn’t scale well as our team grew. Increasing the number of high-quality ideas ready for experimentation is one of the biggest levers for increasing the impact of a growth team, but our leads have less and less time to research and find great opportunities as we continued to scale the team.

To address this, we changed the structure of our Growth team to be bottom-up, meaning everyone — engineers, product managers, engineering managers, designers, etc. — is expected to contribute quality experiment ideas. However, this model immediately runs into the reality that finding great growth opportunities is a rare skill that usually only comes with experience. We developed Experiment Idea Review (EIR) as a way to quickly train team members to find high-quality growth opportunities and continuously build our idea backlog. The rest of this post describes how to run EIR effectively.

An inversion of the traditional growth team model for experiment ideation. Dotted lines represent the ongoing growth of the team. Enabling everyone to generate high-quality experiment ideas scales much better than the top-down model.

Experiment Idea Review Guide

What is Experiment Idea Review?

In a nutshell, EIR is a recurring meeting to pitch ideas. It has two main goals: (1) train team members how to come up with quality ideas and (2) build a sufficient backlog of high quality experiment ideas. EIR can be run at various intervals (e.g., weekly, monthly) depending on how many quality ideas the team needs for a healthy backlog, but the more practice everyone gets, the better they become at originating high-quality ideas.

It’s important to note EIR is not a brainstorm. At Pinterest, we historically used brainstorms to generate growth experiment ideas. We would get everyone together in a room and have each person come up with ideas off the top of their head, which we would then try to feed into our roadmap. However, we’ve learned simple brainstorms tend to only be effective when the goal is to come up with solutions to very specific, scoped problems (e.g., the email open rate for dormant users is good, but the clickthrough rate is low — how might we solve that?). People are able to contribute much more when given sufficient time to research ideas ahead of time versus just coming up with ideas in the moment.

Experiment Idea Review Process

Before an EIR meeting, team members come up with ideas and spend up to an hour filling out an experiment idea doc. A well-written document quickly provides readers with enough detail to make an educated guess of the idea’s potential value, so the doc should be as detailed as possible.

An experiment document should contain the following components:

  • Problem statement:What user problem is being solved? (we always want our experiments to benefit users)
  • Screenshots & videos: What does the current user experience look like?
  • Experiment idea:What are the details of the proposed experiment’s design, setup, and goals?
  • Hypothesis:What will the experiment result in, and how will it significantly improve key metrics?
  • Opportunity size:How many users per day will see the experimental feature?
  • Estimated impact:Based on the opportunity size, what is the expected range of impact?
  • Investment:How much engineering/design/etc. time would be required to invest into this idea?
  • Precedence:Are there previous related experiments that our company or other companies have run? What can we learn from them? If we’ve run this type of experiment before, why will it work differently now?
  • Recommended Rating: Based on all of the above, should we work on this experiment immediately, put it in the backlog, or not prioritize it?

During EIR, you should plan on spending around five minutes for presentation and five minutes for discussion per idea. Leads should give extensive feedback about the idea, and the discussion should conclude with action items. Different teams can use different rating systems, but in general each rating should match an action item. For example, good action items are “work on immediately”, “move to backlog”, and “deprioritize”.

Best Practices

We’ve run into a few issues making EIR effective, and we’ve developed a few key best practices in response to them:

Problem: People don’t bring ideas
Solution: Create a schedule with assignments for specific people to bring ideas on specific dates. Managers need to set expectations that finding great ideas is required to succeed on a growth team. Consider pairing new and experienced team members so they can work on ideas together.

Problem: People don’t know how to come up with ideas
Solution: Teach people during onboarding how to find ideas through competitor audits, user flow walkthroughs, metrics deep dives, and examining past experiment results. Build a library of high-quality idea docs to serve as examples for new team members to learn from.

Problem: Idea quality is lacking
Solution: Note that this is expected when ramping up an EIR process. To accelerate quality improvement, leads should give significant feedback after every idea (there’s a feedback guide later in this post to help with what kind of feedback to focus on). Also, leads should emphasize that one good idea will generate more impact than ten bad ones and will take much less time to implement as well. It’s also important to set expectations around the amount of prep time that should go into researching an idea; typically, we recommend spending between 30 minutes and two hours prepping an idea for EIR.

Problem: Idea document is not detailed enough to give a rating
Solution: Have the leads collect the idea one-pagers ahead of the meeting so they can scan through and spot if any ideas are lacking sufficient detail. If they find any, ask the authors to flesh their ideas out, and reschedule the ideas for review in a later EIR meeting to avoid spending review time twice for the same idea.

Problem: Good ideas are not acted upon
Solution: For really strong ideas, assign owners to work on them right there in the meeting.

Feedback Guide

Here are some common feedback areas for helping team members improve the quality of their experiment ideas:

  • Opportunity size too low:If too few users see an experimental feature, the impact will be limited.
  • Impact estimate unlikely:It’s common for people to overestimate how much impact a proposed change may have.
  • Precedent not listed:New team members usually aren’t familiar with past experiments, so it’s good to share precedent learnings early and often.
  • Precedent listed without discussion:“Why will it work differently now?” should be answered for any listed precedents.
  • Investment size incorrect:People tend to underestimate how long projects will take. There is significant team overhead for each experiment, so you can’t just consider development time.
  • Ratings too lenient:Look at your past experiments and divide the amount of big wins (not including shipped but small win) by the total number of experiments. It’s likely a low percentage, which should encourage people to be critical about what experiments to commit time to.

For more information, watch the talk Jeff Chang gave at the SF Growth Engineering Meetup about bottom-up growth teams and experiment idea review:

Our Results

After running the EIR process for over a year, we’ve seen the following benefits across our Growth team:

  • Depending on the subteam, 50–100% of experiments are ideated by non-lead team members
  • We’ve scaled our Growth team to 100+ members while continuing to increase the overall experiment impact of each subteam
  • New team members are more effective since they’re able to more quickly learn about what the team has tried in the past and own new experiments with increased confidence.

Experiment Idea Review has helped the Pinterest Growth team scale its impact, and team members have enjoyed increased agency in deciding what projects they work on while also having more direct influence on their own personal impact and career growth. If you are an engineer, PM, analyst, or designer who wants to join a high impact bottom-up growth team, please check out https://pinterest.com/careers.

Thanks to the many people on the growth team who helped test and improve this process!

Originally published on the Pinterest Engineering Blog

Managing Your Growth Team’s Portfolio: A Step-by-Step Guide

Managing a Growth team in many ways can be like managing an investment portfolio. Each individual experiment or project is an investment and the goal is to maximize the long-term return (i.e. impact) of your portfolio. Having worked in Growth for almost 10 years I’ve seen how the portfolio of a Growth team can evolve over the course of several years. In this post I’ll cover the three major classes of projects that a Growth team works on and how Growth teams should be adjusting their portfolio allocation over time based on changing needs.

Three Asset Classes in Your Growth Portfolio

Growth portfolios are primarily made up of the following three “asset classes”:

Iterative Experiments – This is what Growth teams are best known for. Iterative experiments are A/B experiments in an established area. Examples could be changing the color of a button, building a new email type, etc. Typically, the average success rate for iterative experiments can range from 30% to 70% and often depends on the domain the team is working in.

Investments – These are either tech investments or user experience investments. Tech investments should increase the capacity of the team to drive more impact in the future. Examples include building a new tool to automatically localize copy experiments or refactoring a major piece of code to run experiments more easily in that area in the future. User experience investments are investments to make the product experience more delightful without the expectation of driving growth (or possibly even slightly hurting growth). The goal of user experience investments is to strike a balance between trying to drive metrics and protecting the brand reputation. Examples could include improving an email unsubscribe flow to make unsubscribing easier and clearer.

Big Bets – These are what they sound like – big bets to open up new opportunities. They take a lot of effort and have a lower probability of success, 20%-40%, but can be very impactful if they work. Examples are building out new growth channels or a complete overhaul of your new user onboarding flow.

Increasing Your Portfolio Returns

One of the most important ways to increase the return on your portfolio is picking the right investments to make (i.e. project selection). You have limited capital (i.e. engineering time) so it is critical to make wise investment decisions. When it comes to improving project selection there are two big levers:

Lever 1: Increase the rigor of your project selection process

In his seminal book on management, High Output Management, Andy Grove, former CEO of Intel states that a common rule we should heed is to detect and fix any problem in a production process at the earliest stage possible. In Growth, every experiment you work on has an opportunity cost. Every failed or low impact experiment you run was time that could have been spent working on a different experiment that maybe could have had a better shot at being successful. On the very first Growth team I was a part of, the way we selected projects was very democratic. We brainstormed a bunch of ideas, wrote them up on a whiteboard, and then people voted on which ones we thought we should do. What we later realized was that this is one of the worst ways to select projects because people vote without any data and just go on gut feeling. In hindsight, we realized that people were voting to work on flows that weren’t perfect and that they felt confident could be improved. However, the reason why those flows weren’t perfect was because they were often not that important or impactful. In Growth it is usually more impactful to make a small incremental improvement at the top of the funnel of a core flow than to fix a broken less used flow.

To improve how we selected projects, we introduced a much more rigorous process and started to calculate the expected value of the project per week of engineering work and picked the ones with the highest estimated ROI.

Expected Value = probability of success * # new active users / amount of engineering work

You are making an estimate for each part of this equation, but it is still helpful in comparing different projects. For estimating the # of new active users, it is helpful to not just pluck a number out of the air. You should start with a hypothesis on what metric you will move and by how much and then model out how that impacts your number of active users. For instance, if you think a new email subject line will increase the open rate by 5%, a rough way to model the number of incremental new users = # of clicks from the email * percent of clicks coming from dormant users * 0.05.

Lever 2: Increase the number of high quality project ideas

Once you have a good process to select projects, the second lever is to increase the pool of good projects you have to select from. To do this you need to make sure it is not just the Product Manager’s job to come up with ideas for the roadmap, but instead build a bottom-up culture where it is the job of every single person on the team to come up with ideas. That means every engineer, every designer, every analyst, should be required to contribute experiment ideas. When it is everyone’s job to contribute to experiment ideas, the number and pipeline of experiment ideas will dramatically increase. With proper training and feedback you can ensure that those ideas are high quality.

Lever 3: Maximize impact of every experiment you ship

The final lever is to make sure you are really maximizing the impact of every experiment you ship. The way to do this is related to lever 2 of building a bottoms-up culture. Every engineer should have ownership over experiments they are assigned to work on, even if it wasn’t originally their idea. What ownership means is that they should be striving to figure out how to make the experiment as impactful as possible by improving on the initial idea or by coming up with additional variants to test. Doing this will help make sure you are squeezing as much juice as possible out of every experiment. If a team of 10 engineers works on 120 experiments/quarter there is no way a single person can think deeply about each and every one of the 120 experiment ideas. Taking a bottoms-up model where the people spending the most time working on an experiment idea are empowered to come up with suggestion and figure out how to maximize that idea will help make sure you’re not leaving any money on the table.

Evolving Your Portfolio Over Time

While it is important to try and maximize your portfolio’s returns, it is also important to understand how the portfolio makeup needs to adjust over time. Growth teams typically go through three stages and how they allocate between different project classes should evolve with what stage they’re in.

Startup Stage:

For a Growth team at an early stage startup or a Growth team working in a brand new area at a mature company, there are a lot of unknowns. Not much has been tried, there is a lot of low hanging fruit, and a lot of foundational stuff might be missing. For instance, you might not have the dashboards you need or you might not even have an experimentation framework in place at an early stage startup. However, the team can’t just focus on investments the entire quarter. The team also needs prove to leadership that they can drive results and show they can move business metrics up and to the right.

At this stage you might want the following allocation:

Iterative Experiments (33%) – You will want to get some early quick wins to prove to leadership that the team can drive results. This will build confidence in the team and give you the space to work on projects that will pay off in the long term.

Investments (33%) – There are typically several foundational investments that need to be made early on to set up the team for long term success. It could be setting up logging pipelines, reports, etc.

Big Bets (33%) – Because there are so many untapped areas, the team needs to take a stab at a couple big rocks to start exploring the problem space, see what pays off, and start to form a longer-term strategy.

Growth Stage:

Typically, after 2-3 quarters a team has gotten some traction in the area and has started to figure out what works. This is where they enter the growth phase, where they typically really focus in on iterative experiments and driving impact. The growth stage lasts many quarters or years and this stage embodies what a stereotypical Growth team looks like.

Iterative Experiments (70%) – This is where the bulk of the team’s impact will come from.

Investments (15%) – Continue making investments to increase the team’s ability to drive long term impact.

Big Bets (15%) – Continue making some big bets to open up new opportunities for the future.

Mature Stage:

Finally, after a team has worked on an area for a long time and has spent many quarters, or many years picking off the low hanging fruit you can start to reach a local maxima where the team has really milked a strategy or an area for all its worth. You’re starting to reach a point where you are faced with making a call on if you should pause working on this area entirely or if it is time to try and break out of that local maxima.

Iterative Experiments (30%) – You should significantly ramp down how much time you’re spending on iterative experiments, but you probably still have some backlog of experiment ideas you still want to work through.

Investments (10%) – Pause most work on investments unless they are tied to the big bets.

Big Bets (60%) – Big bets ramp up significantly to try and find new opportunities and break out of the local maxima. When in a local maximum, it is important to note that you probably have spent quarters or even years optimizing what is now the current control experience. Your first stabs at a radically different approach likely won’t beat control off the bat and you need to be prepared to spend time doing iterative optimization on the big bet to see if it has the potential to eventually surpass control. If one of these bets open up a whole new area, the team might transition back to the Growth Stage.

Conclusion

Take all the percentage allocations in this post as just a rough guideline. However, every leader in Growth should be actively thinking about and managing their portfolio and their team processes that produce that portfolio. Make sure you are maximizing your return through really assessing and improving your project selection process. Build a culture that not only empowers, but requires, everyone to think about how to help their team hit its goals. Finally, understand what stage you are in and look ahead, out in front of the team, to anticipate how your portfolio allocation needs to change to ensure the team’s long-term success.

 

Increase funnel conversion with Psych

Every year engineers on the Pinterest Growth team organize an internal conference called Growthcon. The goal of Growthcon is help share learnings, insights, and best practices from across all the teams in Growth. Every Growthcon has a mix of talks from internal speakers, breakout learning labs, and a keynote speaker. For this year’s keynote we invited Darius Contractor, who has led Growth at Airtable, Facebook Messenger, and Dropbox, to come give a talk about his Psych framework for funnel optimization.

Darius dissects flows from Match.com, Hooked, Airbnb, and Airtable and walks through step by step how you can use the psych framework and user psychology to understand what will make a user convert through a flow. Watch the video below to learn about how you can apply the psych framework to your conversion rate optimization efforts.

 

Measuring Incrementality For Emails @ Netflix

Chris Beaumont, a Data Scientist at Netflix, recently gave a talk at the SF Growth Engineering Meetup on a novel new approach Netflix took to continuously measure and understanding the incremental impact different emails had on subscriber growth. Chris expands on the original blog post he published to go more in depth on how Netflix created an automated system to do per-type holdouts and how they analyzed the causal impact each email and push notification had on different business KPIs. He also discusses how this approach is an improvement upon traditional A/B testing or just monitoring click and open rates. Watch the video to learn more.

 

5 Principles For Goaling Your Growth Team

There are tons of posts on how to set a Growth team up for success from a hiring perspective, process perspective, tooling perspective, etc. However, one of the most important things to get right is setting up what each team is goaled on. This is especially true as a Growth Org starts to scale with multiple teams working in different areas, developing their own strategies, and each team figuring out how to drive their metrics up and to the right. It becomes critical that the goals and metrics are architected in a way to ensure each team is building towards a long term sustainable business.

Casey Winters wrote a great post on 5 mistakes to avoid when setting goals for a Growth team and why you should set goals on absolute numbers instead of percentages. In this post, I want to build on that and share 5 guiding principles I’ve found to be effective when architecting goals for a team, ensuring the team is aligned with the rest of the company and setting them up for long term success.

 

Principle #1: Isolation

The first key ingredient for a team level goal is making sure that it is a metric that the team can directly influence and has some degree of isolation from other teams, seasonality, etc. Team level goals should be a yardstick against which the team can measure progress. However, if that metric can be heavily influenced by other teams, by seasonality, or other factors outside of the team’s control, it starts to lose value as a way for the team to measure its impact and progress.

At Pinterest, the way each team in Growth measures its progress is by calculating the absolute number of incremental Weekly Active Users added by every A/B experiment run and summing up that impact over the quarter. By using A/B experiments to measure each team’s impact, the A/B experimentation framework provides isolation between teams, even when teams work in overlapping areas. The one downside of summing up experiment impact is that it has a tendency to overestimate the impact. Another approach is having team level holdouts for every experiment they run in the quarter, which can result in a more accurate measure of a team’s impact, but at the cost of significant additional engineering maintenance and overhead.

 

Principle #2: Avoid Misalignments

The second key ingredient of a team goal is making sure it does not put you at odds with other teams in the company. I recall on one occasion speaking to a Growth PM and they were complaining that their team goal was to grow the total number of registered users with the app installed. On the surface, this might seem like a fine metric. The problem was that this metric put them directly at odds with the Notifications team. This is because any time the Notifications team would send a push notification, some small portion of users will delete the app. Misalignments like this suggest that optimizing that metric might not correlate 100% to business success.

One possible way to overcome this is by having multiple teams in Growth all work to drive the same metric while defining swim lanes and areas of ownership to ensure each team has a clear charter. For instance, at Pinterest, most teams in Growth goal on Weekly Active Users, and each team has clear areas they own such as Emails & Notifications, or Signups & Logins.

 

 

Principle #3: Tied to Long Term Success

            The third key ingredient of a team metric is tying it to long-term business success. Growth teams are hyper metric focused and even with the best intentions, they can sometimes over optimize for a metric. To counteract this, you want to ensure that metrics cannot easily be “gamed” and that they are tied to the long-term success of the company. You can do this by going deeper than the surface level metrics and setting goals based on down funnel engagement.

A simple example is signups. It is natural for a team focused on user acquisition to perhaps goal on signups as the metric they drive. However, signups are a metric that can relatively easily be juiced in ways that might not lead to the best long-term outcomes. For instance, you can remove steps from the signup flow to get more users through the signup funnel, but those steps may be important in significantly improving the user experience once the users are in the product. Or perhaps, the acquisition team starts driving a ton of signups from low-quality traffic sources and signing up a ton of users that end up not sticking around. A better metric for a team focused on acquisition might be a metric like activated signups, where they only count signups that result in a long-term retained user. In this way, the team is ensuring the metric they are trying to drive is always aligned with long term business success.

 

Principle #4: Guardrails

Even with metrics that are tied to long term success, it can sometimes be necessary to setup guardrails to protect the user experience. A great example is emails and notifications; sending more emails always lifts engagement. Even if those additional emails result in more unsubscribes, those unsubscribes never overcome the lift in active users from just sending more. Intuitively though, everyone knows that there can be a long-term cost to sending too many emails which can cause user fatigue or brand perception. The problem is the long-term cost doesn’t appear in days or weeks, but over years.  To help protect the user experience, the team needs to establish guardrail metrics around unsubscribe rates and spam complaint rates to help protect the user experience while they drive towards their goal. The guardrail metrics would ensure that optimizing their northstar metric of active users isn’t at the expense of user experience.

Guardrails can also be helpful as a cross-check for the main metric the team focuses on to help catch potential issues. For instance, if the Emails & Notifications team decides to measure its impact through A/B experiments, it is important to setup a cross-check metric for the team to track the health of the channel outside of experiments. If a particular ISP starts marking the email as spam, it could significantly impact the business and the team needs to be able to catch that. The team would need to setup a guardrail metric like daily email clicks, or even better, daily engaged sessions from email, to ensure the team can catch and identify issues that fall outside of experiments.

 

Principle #5: KISS

            Finally, there is a well-known principle in Computer Science called KISS, which is an acronym that stands for “Keep it simple, stupid.” This principle was originally created by the Navy and states that most systems work best if they are kept simple rather than made complicated. With metrics, it can be easy to get a bit carried away and create extremely complex metrics or have several different metrics that a team is responsible for driving. When designing team metrics or a team goal, try not to overcomplicate things and just keep it simple, stupid.

The 27 Metrics in Pinterest’s Internal Growth Dashboard

One question I often get asked by people starting out on growth is “what metrics should be in my growth dashboard?”. I’ve written before about what metrics we value at Pinterest. In this post however, I’ll give people a peek behind the scenes and share what our internal growth dashboard looks like.

We have organized our dashboard to reflect our user growth model. We start with our top line growth metric of MAUs. Then we follow the user lifecycle funnel; starting with acquisition metrics, followed by activation, engagement, and finally resurrection.

Pinterest Growth Model

 

MAUs

1. Current progress to goal: Current number of MAUs & how much progress we’ve made towards our quarterly MAU goal.

2. MAU Forecast: Forecast of the number of MAUs we could expect to have extrapolated from our growth rate at the same time during previous years. We include this metric to help us anticipate the effect of seasonality on our growth numbers.

3. MAUs by app

4. MAUs by gender

5. MAUs by country: Tracking total number of MAUs in every single country would obviously be overwhelming to view on a chart, so instead we bucket countries together. The buckets we use are USA, Tier 1, Tier 2, Tier 3, and Rest of World. The tiers are based on size of Internet population, Internet ad spending, etc.

6. MAU Accounting: The MAU accounting helps us see what factors are contributing the most to our MAU growth. Specifically we split out total number of signups, resurrections, existing users churning out and new users churning out.

Growth MAU Accounting

 

Acquisition

7. Total signups

8. Signups by app

9. Signups by referrer

Signups by referrer

10, 11, 12. Invites Sent, Unique Invite Senders, and Invite Signups

 

Activation

 13. Overall Activation Rate: 1d7s is a term we use to refer to users who come back 1 or more times in the week following signup. We measure overall activation rate as 1d7s/signups, or in other words, the percentage of new signups that visit Pinterest again in the week following signup.

 14. Activation by app: This is the same metric of 1d7s/signups split out by platform. We’ve seen that different platforms can actually have pretty dramatic differences in activation rates.

15. Activation by referrer source

16. Activation by gender

Activation rate by gender

17. Overall signups to 1rc7: This metric is similar to the signups to 1d7 except it measures the percentage of new signups that repinned a pin or clicked on a pin in the week following signup. We use this metric to measure as a leading indicator of how well we are activating users into the highly engaged user buckets

 18. 1rc7s by app

19. Signups to 1rc7 ratio by app

20. Signups to engagement funnel by app: This metric tracks the percentage of new signups that are still doing key actions during a one-week time window of 28-35 days after signup. Specifically, we track 35 days after a user signs up, what percentage of them are still an MAU, WARC (weekly active repinner or clicker), WAC (weekly active clicker), or WAR (weekly active repinner).

21. Signup engagement funnel by gender

22. Signups to WAU 35 days after signup: This is one of our key activation metrics. We track the total percentage of users who are still a WAU one month after signup.  Specifically we look to see what percentage of signups were active between 28-35 days after signup.

 

Engagement

 23. *AU ratios: We track the ratio of DAUs to MAUs, WAUs to MAUs, and DAUs to WAUs. The ratio between *AUs is a popular metric to gauge how engaged users are with your app.

*AU ratios

24. Email Summary by type: Table of total number of emails sent, opened, & clicked-through split out by email type.

25. Push notification summary by type: Table of total number of push notifications sent & opened split out by type and by platform (iOS & Android).

 

Resurrections

26. Resurrections by platform: Total number of users that were dormant for 28+ days, but then came back to Pinterest, split out by which platform they came back on.

27. Resurrections by referrer

Retention rates by referrer

To wrap up, you can see we put a big emphasis on activation (the process of getting a new user to convert to a MAU). This is because we consider activation critical to long-term sustainable growth. Strong activation rates are necessary if you want to be able to scale a service to hundreds of millions of users.  We also put an emphasis on segmentation by gender, country, referrer, etc., to more deeply understand how different segments of users interact with Pinterest and see which segments are underperforming. If you have any questions feel free to ping me on twitter or drop me a line.

4 Metrics Every Growth Hacker Should Be Watching

The metrics typically advertised by startups are total users, daily active users (DAU), and monthly active users (MAU). While these numbers might be good to share with the press, they are only vanity metrics because they don’t give any real insight into your growth rate or the quality of the users you’re bringing in. Here are 4 metrics you should really be paying attention to if you’re trying to drive sustainable user growth.

Daily Net Change – Daily net change tells you on a daily basis how much you’re user base has grown (or shrunk). In a single graph you can assess new user acquisition, re-engagement, and retention and can easily see the impact of each component on your current growth rate.

Here is the breakdown of the different components and how they are calculated:

New Users: how many new users joined the service today?

Reactivated Users: how many existing users used the service today for the first time in 28 days?

Churned Users: how many existing users last used the service exactly 28 days ago?

Net change: new + reactivated – churned

Net Change Graph

Growth Metric: Net Change in Users

 

Core Daily Actives – The problem with the daily active user metric is there is not concept of quality users or retention. You will often see DAUs jump from a user acquisition campaign, but it is impossible to tell from the metric if those users are immediately dropping off or if they are sticking around. Core Daily Actives rises above this noise by only counting users that have been using your service on a regular basis. To get this metric, you calculate the number of users that used your service today who also used your service 5 or more times in the past 4 weeks. This metric is much more useful than DAUs because it focuses on the bottom line: growth of repeat users.

Cohort Activity Heatmap – The cohort activity heatmap is by far my favorite because it is the most insightful metric on this list. What the metric shows is how your user retention curve has changed over time. It can be a bit complex to read at first because so much data is crammed into a single graph, but it is very powerful once you know how to use it.

This is how you interpret the graph:

  • The unit of the x-axis is days and each column corresponds to the group of users that joined on day X (each group is called a cohort)
  • The width of a column on the x-axis represents the size of the cohort (i.e. the wider the column, the more users joined on that day)
  • The unit of the y-axis is also days and each row represents Y days after the cohort joined the service. The bottom row of the graph represents day 0, the very first day the user joined the service, and the top row represents day 59.
  •  The color of each rectangle represents activity level. It is calculated by determining the percentage of users that joined day X and used the service on day Y. The scale ranges from red for a high percentage to blue for a low percentage.

 Cohort Activity HeatmapGrowth Hacker Metric: Cohort Activity Heatmap

 

Conversion Funnel – The final metric is tracking the conversion funnel for flows that affect new user acquisition, retention or re-engagement. A conversion funnel is simply splitting up a process into its constituent steps and tracking how many users make it through each step. This metric is widely used, but it is common to analyze the conversion funnel for a flow once or very infrequently. Really, the conversion funnel for important flows should be tracked on a daily basis for adverse changes because even a difference of a few percentage points can compound over time.

This is an example of the conversion funnel for inviting users to a service.

1)   How many users saw the invite prompt?

2)   What percent of users clicked on the invite prompt?

3)   How many invites were sent per user that clicked on the prompt?

4)   What percent of invites were viewed?

5)   What percent of invitees clicked on the link in the invite?

6)   What percent of invitees that clicked on the link, joined the service?

7)   How many new users joined the service as a result of the invite.

 

This is just a sampling of some of my favorite growth hacking metrics. There are many others and usually the best metrics for you depends on your situation. If you know any metrics that you think should have been on this list, please drop me a line.

The Secret to Effective Geofencing

You’ve probably heard of geofencing, but in case you haven’t, geofenced notifications are alerts that users receive on their smartphone when they are near a particular location. They are a powerful activation & re-engagement tool for location based apps because they communicate information to a user at a time that is contextually relevant. Over the past year, geofenced notifications have become popular with many shopping apps such as RetailMeNot, RedLaser, etc. In this post I’ll cover some of the basic problems you might encounter with developing geofencing.

1. Inaccurate Location Data
The dirty little secret of most location data providers is that their data just isn’t that accurate. Typically for geofencing you will want to geofence a small radius (ex: 150 meters) around the point of interest. The way these data providers get latitude & longitude information is through a method called address interpolation. The problem is that the latitudes and longitudes from this method can often be off by hundreds of meters. This means when a user is standing directly at the point of interest, there is a good chance they will not fall into the geofencing radius! To overcome this problem you either have to manually fix the location data (like eBay did) or use user data to automatically correct the locations.

2. Battery Drain on Android
Android has built in support for geofencing, but unfortunately it has severe battery drain issues on certain versions & devices. To work around this issue, many developers have taken to developing their own proprietary geofencing framework based on polling GPS. However, a lot of attention needs to be given into tweaking the framework to ensure users receive notifications in a timely manner, but without significantly draining their battery. Some ways to decrease battery drain are to poll less frequently, or instruct the device to only use wifi data to get a location fix rather than turning on the GPS.

3. iOS’s hard limit on the number of geofenced regions
Thankfully, iOS has a geofencing framework that doesn’t seem to drain user’s batteries. Unfortunately, iOS sets a pretty low limit on the number of regions you can monitor. This means to geofence effectively, you need to update which regions your geofence as a user moves around. Luckily, iOS has another framework that will wake up your app when the user has moved a significant distance (on the order of 5-10 miles). You can use this framework to wake up and update the regions your app is geofencing.

These are just a few of the problems that might be encountered when developing geofencing, but geofencing is such a powerful engagement tool and it is well worth the effort.

3 Lawsuits Every Growth Team Needs To Know

I’ve found that people in Growth are all too often not aware of some of the major laws that govern the industry. In this post, I’ll cover three major lawsuits that sued over growth tactics used by well-known companies and some principles for how your Growth team can stay out of trouble.

 

Hickey v. Voxernet LLC

In 2012, Voxer was sued over their invite flow where users sent SMS invites to their friends. Instead of just popping up the SMS app with a pre-populated message, the flow in Voxer was it would popup a contact list picker and have the user select a number of different friends from their contact list to invite. After the user selected the friends they wanted to invite then Voxer sent those friends an invite from their servers. The claim was that Voxer sending SMS invites violated a clause in the Telephone Consumer Protection Act (TCPA) against Automated Telephone Dialing Systems (known as ATDS or auto-dialers). Voxer ultimately settled, and in the three years following that lawsuit, a number of lawsuits were filed against other companies that also had similar flows which used a contact list picker to allow users to send multiple SMS invites [Sterk v. Path Inc.][Huricks et al v. Shopkick, Inc][McKenna v. WhisperText LLC][Glauser v. GroupMe, Inc.][Reichman v. Poshmark, Inc]. A lot of the lawsuits stemmed from some ambiguity on what was considered an auto-dialer. Then in 2015, the FCC issued a ruling to clarify the ambiguity and laid out clear guidelines for when SMS invites were ok. Namely, they needed to meet the following criteria:

  • The invites required human intervention (i.e. they had to be initiated by the user) The user needs to play a role affirmatively deciding to send a text message, selecting who to send to, and preferably controlling the contents of the text message.
  • It had to be clear to users that their friends were going to receive a text message and how many messages would be sent as a result of their action (as opposed to a different communication channel such as an email).

The biggest takeaway is that while sending SMS invites can be very powerful, they can also open a company up to a lot of legal risk. To mitigate that risk, be very thoughtful about how you design your invite flow and preferably get a lawyer to carefully review your invite flow and provide a written legal opinion with screenshots of the flow.

Poshmark SMS invite

Perkins v. LinkedIn Corp.

LinkedIn was sued in 2013 for sending invite reminder emails. When a LinkedIn user invited their friends or colleagues to the service, LinkedIn would send an email to those people. If they didn’t respond to the initial email, they would receive a 2nd and a 3rd email as a follow up reminder to join the service. While on the surface, that doesn’t sound very nefarious, the lawsuit alleged that users had consented to the 1st email, but they had not consented to the 2nd or 3rd reminder emails. The lawsuit also alleged that by sending reminder emails the user had not consented to, it could potentially damage the user’s reputation by spamming their contacts without their consent. It also alleged that users had not consented to their name and profile picture in the reminder email. LinkedIn ultimately settled for $13M, changed its product policy and privacy policy. The takeaway from this lawsuit is that even for email invites it needs to be very clear to users how many people will be contacted and how many times their friends will be contacted.

LinkedIn Invite Reminder Email

 

Opperman et al v. Kong Technologies, Inc. et al

In 2011 and 2012, many apps such as Instagram, Twitter, Yelp, Path, Kik, Gowalla, Foodspotting, and Foursquare were caught uploading users’ contact lists to their servers without user knowledge or consent. In most cases, the reason these companies were doing this was to do friend recommendations (to either identify friends already on the service or to recommend friends to invite). Apple quickly added the Contact List Permission to iOS 6 as a result of the scandal and the defendants agreed to settle the class action lawsuit for $5.3M. The takeaway from this lawsuit is that even if something is technically possible, and even if you are allowed by your products Terms of Service, you should be very clear and upfront when collecting data that a user would not reasonably expect be collected.

iOS 6 contact list permission

Conclusion

Path Inc. in particular got hit with a triple whammy. Not only was it sued for SMS invites in Sterk v. Path and had to pay into the Opperman et al v. Kong Technologies, Inc. et al settlement, it was also fined $800,000 by the FCC for violating the Child Online Privacy Protection Act (COPPA). The violation stemmed from the fact that a few thousand children under age 13 were allowed to sign up for the product and Path collected personal information from those users without parental consent.

Growth teams and startups can have a mentality of “move fast and break things.” However, it is important to understand what areas to pay careful attention to if you want to avoid the company getting bogged down in lawsuits and legal wrangling: SMS, invites, referrals, and data privacy. These two principles can help you stay out of trouble:

  • Make sure the Growth team is aware of major laws that impact their work like the Telephone Consumer Protection Act (TCPA), Child Online Privacy Protection Act (COPPA), European General Data Protection Regulations (GDPR), etc.
  • Make sure you always get informed user consent any time you are collecting data the user might not expect, or when the user is going to send an invite/referral.

 

Disclaimer: I am not a lawyer and this post in no way constitutes legal advice

Growth Engineering Meetup (Airbnb, Pinterest, Twitter)

Pinterest recently hosted a SF Growth Engineering meetup at the Pinterest office to discuss how some of the most successful companies drive growth at scale. See the video below to hear what speakers from Pinterest, Airbnb and Twitter had to share.

Synopsis of the talks

Paid growth @ Airbnb

Ganesh Venkataraman discusses the various channels and techniques used towards paid marketing at Airbnb as well as the underlying challenges around data/machine learning and attribution.

Speaker: Venkataraman leads guest growth engineering at Airbnb. Previously he worked at LinkedIn where he led careers relevance (job search, recommendations and statistical insights on salary). He has over 20 publications/tutorials/talks in leading conferences including SIGKDD, SIGIR, CIKM, SODA etc. He also has filed 16 patents in areas related to search and machine learning.

Onboarding component framework @ Twitter

Michael Lin covers Twitter’s Onboarding Component Framework which allows backend systems to construct personalizable signup flows using generic client-side components. This talk addresses the infrastructure, experiments and learnings from building this system.

Speaker: Lin is an infrastructure engineer on the Onboarding Team at Twitter. During his four year tenure at Twitter, he has also worked on account security, anti-spam and ads infrastructure. Michael is a graduate of the University of Toronto.

Keeping your resurrected users out of the graveyard @ Pinterest

Pinterest has millions of dormant users that resurrect each week, but often times they churn back out. Allan Blair discusses how Pinterest has built a strategy to classify, measure impact and ultimately increase our resurrected user retention.

Speaker: Blair is a full-stack engineer on the Engagement Growth team at Pinterest. Previously he worked as an Investment Banker at J.P. Morgan on Wall Street. Allan is a graduate of Rensselaer Polytechnic Institute in upstate New York.


The SF Growth Engineering Meetup holds these talks like these about once a quarter. Feel free to j. Join the SF Growth Engineering Meetup Group to stay in the loop and find out about future events.