When do features drive growth?

As I mentioned in my previous post, I often see this belief in product development that adding new product features to a product will help spur growth. The thinking behind it is basically that more features == more value == more growth. I disagreed with that thinking in my previous post and received a few questions about that point, so I will expand upon it in this post.


First I want to define what a core product feature is. A core product feature is a feature that is part of the normal everyday use of the product. So, I’m excluding typical growth features such as new user flows, invite referrals, sharing to social networks, SEO, etc., where the immediate impact on growth is very clear, but the feature is not part of the core usage of the product. My hypothesis is that new core product features only help change a company’s growth trajectory if it creates an engagement loop or creates a step-change improvement in the amount of value the average user gets from the product (preferably it does both). Specifically, for a feature to accelerate growth, I think it needs to meet the following criteria:

A) Most important is mass adoption by the user base. Over 50% of the users will need to interact with the feature on a regular basis (i.e. daily or weekly basis).

B) The feature creates an engagement loop that allows you to email or notify users on a regular basis. The content in those emails/notifications also needs to be compelling enough that it maintains a high click through rate over time.

C) Or as an alternative to B, the feature is a step-change improvement in the core value of the product for a majority of the userbase.

Facebook: A Case Study 

To put this hypothesis to the test, Facebook serves as a great case study of which types of features drive Growth and which do not. Over the years, they have launched Photos, News Feed, Platform, and Chat, which have all been major drivers of growth and engagement. However, they have also launched Questions, Places, Deals, Gifts, and Timeline, which have not fared as well in terms of driving growth and engagement.

To define what Facebook’s core product value is, I’ll use one of their more succinct mission statements from 2008, which is: “Facebook helps you connect and share with the people in your life.” [1]

Features That Did Drive Growth and Engagement

Photos: Photos was launched in 2005 and was one of the first major new features Facebook added after launch. Facebook has stated before how successful the Photos product was at driving engagement. If we look through the lens of the criteria laid out above, we can start to understand why.
A) The majority of users upload photos or are tagged in photos.
B) Tagging allows Facebook to re-engage users and has been so successful that Facebook has heavily invested in facial recognition to make tagging easier.
C) Photos provides a significant increase in the core product value by allowing people to much more easily share important moments and events in their lives and allowing people to be much better connected with what their friends and family are doing.

News Feed: News Feed was very controversial when it first launched in 2006, but ultimately led to a significant uptick in user engagement. Looking at our criteria:
A) Pretty much 100% of users view their News Feed.
B) News Feed dramatically improved the content sharing engagement loop. After News Feed, when someone shared a set of photos or a status update, those shares had dramatically higher visibility compared to when they were only visible by navigating to a user’s profile. This meant that these posts now received many more likes and comments that Facebook would then notify the user about.
C) News Feed provided another step-change improvement to Facebook’s mission. It was now much easier to passively stay connected to friends by seeing their posts and updates. As previously mentioned, it also fundamentally altered Facebook’s value proposition. It was no longer just a way to keep in touch, but it was now a way to broadcast and communicate with your social graph.

Platform: In a bold move at the time, Facebook opened up their platform to third party developers in 2006 [2]. Although Facebook has since clamped down significantly on the platform in the interest of user experience, Facebook Platform was a big engagement driver for a period of a few years.
A) For the first few years, a significant percentage of users interacted directly with Facebook apps or would get notifications from others using the apps.
B) Facebook outsourced the work of constructing the engagement loop to third-parties. Apps like Farmville, etc. had to create their own engagement loops to survive by using gamification mechanics such as crop harvesting to bring users back. By bringing a user back to Farmville, Zynga also brought a user back to Facebook. They also generated billions of notifications to Facebook users by getting users to spam app requests out to friends until Facebook finally clamped down on the app request spam.
C) Social gaming was the primary category of app that achieved significant traction on Facebook Platform. Those apps did little to improve Facebook’s core product value, so point C is not applicable.

Chat: Messages have been part of Facebook since its initial launch in 2004. However, in 2008 Facebook launched Chat [3]. Examining Chat, we can see that:
A) To ensure mass adoption, Facebook made Chat highly visible by including it as a sidebar overlay on every page.
B) Chat creates a very compelling high-frequency engagement loop.
C) Chat again significantly expanded on the ability for friends and family to stay connected through Facebook and eventually evolved into Facebook Messenger.

Features That Didn’t Drive Growth and Engagement

Questions: Launched in 2010 [4], possibly as a jab at Quora co-founder and former Facebook CTO Adam D’Angelo, Questions never really gained much traction.
A) The number of users using questions rapidly dropped off after the initial launch.
B) Questions did create an engagement loop, but the frequency was relatively low.
C) Questions was an incremental improvement and did not significantly expand on Facebook’s value proposition since people could already ask questions by just posting a status update.

Places: Also launched in 2010 [5], Places was Facebook’s response to the surging popularity of Foursquare. Places promised to enable people to share where they were.
A) Facebook probably did get over half the users to use Places through both checkins and location tagging in photos.
B) Places did not create an engagement loop.
C) Places was an incremental improvement and did not significantly expand on the value proposition with regards to sharing since people could already share where they were through status updates or picture descriptions.

Deals and Gifts: Deals and Gifts are both pretty similar and were aimed more at monetization than growth, but they are still worth covering. Deals launched in 2010 (must have been a busy year) [6]. Gifts first launched in 2007 [7] and then re-launched in 2012 following Facebook’s acquisition of Karma [8]. Looking at our criteria:
A) Fewer than 50% of users bought a deal/gift.
B) Deals and Gifts did have the potential to create an engagement loop via a daily email, but Facebook didn’t capitalize on it. Even if they did, they would have discovered the same thing Groupon and LivingSocial did, which was that click through rates on the emails decay significantly over time as users grow tired of receiving deal after deal (or gift) they are not interested in buying.
C) Deals were not tied to Facebook’s value proposition at all. Gifts at least helped people be more connected, but the frequency was too low and the consumer adoption wasn’t there.

Timeline: Introduced in 2011, Timeline served two purposes. First, it was meant to help users rediscover things they had shared in the past and second, it allow users to more easily share from other apps [9]. However, according to insiders, it did not lead to any significant gains in growth or engagement.
A) Over 50% interact with Timeline.
B) Timeline did not create a significant engagement loop. It generated a lot more posts from apps, but due to the trivial nature of the information being shared, they did not receive many likes and comments compared to organic shares and posts.
C) Timeline was more of an incremental improvement in the core product value rather than a step-change. In terms of being connected to friends and family, sure, you could now more easily scroll through hundreds of posts your friends had made over the years, but no one does that on a regular basis. In terms of sharing, the data shared from apps was trivial information such as a song you listened to or an article you read, but it didn’t necessarily mean you liked the song or thought the article was interesting, so the share had little value.

So What Does This All Mean?

Should we stop all core product feature development that doesn’t meet these criteria? Of course not. Core product teams should continue to develop features to make the product incrementally better and make users happy. However, if you are contemplating adding a new feature on the basis of expecting it to drive growth and engagement, you should ruthlessly evaluate it against these criteria.

Agree? Disagree? Discuss this post on growthhackers.com

Acknowledgements: Special thanks to Casey Winters and Stephanie Egan for helping review and refine this post.

How Can Reddit Solve its Growth Problem?

Reddit has been undergoing a lot of turmoil lately. CEO Ellen Pao resigned ostensibly because she felt she couldn’t deliver the growth numbers the board wanted to see in the next six months. A question was posed yesterday on /r/Entrepreneur about “What would you do as Reddit’s CEO to grow the user base in the next 6 months”. The comments in the post were filled with ideas of new features that could be added or existing features that could be tweaked. For instance suggestions were to tweak the upvote/downvote system, build improved moderator tools, or give greater visibility to underused features such as multireddits, etc

Do Features Drive Growth?

I think one mistake people often make is thinking that new features can help spur growth. They think more features == more user value == more growth. Whether you’re a tiny startup just getting off the ground or a mature product used by hundreds of millions of people, I think new features rarely lead to a significant change in growth trajectory. I believe this is because for a new feature to drive more growth it can’t just add incrementally more value; it has to create a step change in the amount of value that the average user gets from the product.

Reddit has a few rough edges, but they couldn’t have grown to 130 million monthly unique visitors if they didn’t have solid product/market fit & weren’t delivering a ton of value to users already. So, I don’t think adding new features are the answer.

So What Should Reddit Do?

I hypothesize reddit derives a majority of its traffic from core users who by habit check reddit on a daily basis, referral traffic from blogs, and SEO. I think the biggest thing Reddit is missing is an engagement loop to bring non-core users back. Reddit currently does a very poor job of utilizing email, push notifications, and other social media platforms to re-engage users. My guess is because they are worried about being spammy, which is a huge mistake, since these channels can be leveraged in a non-spammy way that actually puts users first.

So what would I do?

1) Re-engagement emails for non-daily active users that give them a digest of the top 10 posts of the previous day or the previous week. I think less active redditors would get a ton of value because it would allow them to discover content they may have otherwise missed.

2) Push notifications for trending posts where timeliness matters (ex: AMAs, breaking news, etc). Often users complain that they discover a post too late after it already has thousands of comments and they feel any comment they make at that point would just get lost in the crowd.

3) Currently Reddit has about 130MM monthly uniques, but only about 9MM registered accounts. In order to make the engagement loop work Reddit would need more aggressive signup prompts for unauth users. Reddit’s user base are pretty anti-signup but I think communicating the value of creating an account would help convince users. Once a user is signed up they can start curating their subreddit subscriptions, they can start engaging in discussions and submitting links (and they can also start receiving the previously mentioned  re-engagement hooks).

4) Finally Buzzfeed, 9gag, the chive, etc., leech a ton of their traffic by repackaging stuff from Reddit and posting it to social media (namely Facebook). Invest in making sharing much more prominent so Reddit can start to capture some of that traffic. This doubles as both an acquisition strategy & a re-engagement strategy.

What would you do?

Share your thoughts in the discussion on growthhackers.com. If anyone from Reddit (or any other startup for that matter) wants some Growth advice, feel free to drop me a line.

Growth Tools & Frameworks

We recently held a meetup at Pinterest’s offices to discuss some of the tools & frameworks that some of the most successful companies have built in-house to enable them to drive growth at scale. We had a great turnout to hear speakers from Dropbox, Pinterest, and Facebook.

A few of the tools & frameworks we heard about were:

– Gandalf, a framework to target marketing messages & campaigns to users

– How Dropbox gets new users to bridge the gap between desktop and mobile

– Copytune, a framework for optimizing copy on a per language basis

– How to build an SEO Experimentation framework

– How Facebook uses “quick experiments” to assess the impact of even the smallest changes, such as bug fixes

If you’re an engineer interested in growth, join the SF Growth Engineering Meetup to find out more about these events in the future.

About the Speakers:

Darius Contractor – Darius works on Growth at Dropbox. He’s previously VP of Engineering at Bebo (acquired by AOL) and PM/Senior Eng at Tickle.com (acquired by Monster). He focuses on building the right product as simply as possible, iterative engineering and having fun. Occasionally, he blogs about psychology at http://darius.com

Viraj Mody – Viraj is an Engineering Manager and has been at Dropbox for 2.5 years where he focuses on onboarding/education/engagement initiatives & building infrastructure for growth. Before Dropbox, Viraj was a founder of Audiogalaxy (acquired by Dropbox in 2012).

John Egan – John is a lead engineer on the Growth team at Pinterest where he leads up efforts on emails & notifications. Prior to Pinterest, he led the Growth engineering team at Shopkick (acquired by SK Planet). You can read his thoughts on growth at http://jwegan.com

Julie Ahn – Julie is a software engineer on the Growth team at Pinterest where she focuses on search engine optimization. She built out the SEO experimentation framework which allows Pinterest to demystify SEO and help drive millions of incremental visits a day to Pinterest. Prior to Pinterest, she was a mechanical engineer in South Korea.

Ran Makavy – Ran is a Director of Product Management at Facebook. He spent the first three years on the Growth team, looking at mobile and emerging market. Today, he is running Facebook’s Local and Entities teams, building consumer products around places and location. Before Facebook, he co-founded Snaptu & grew it to over 100 million active users before it was acquired by Facebook.


Why You Should Be A/B Testing Your Infrastructure

The benefits of using a data-driven approach to product development are widely known. Most companies  understand the benefits of running an A/B experiment when adding a new feature or redesigning a page. While engineers and product managers have embraced a data-driven approach to product development, few think to apply it to backend development. We’ve applied A/B testing to major infrastructural changes at Pinterest and have found it extremely helpful in validating those changes have no negative user-facing impact.

Bugs are simply unavoidable when it comes to developing complex software. It’s often hard to prove you’ve you covered all possible edge cases, all possible error cases and all possible performance issues. However, when replacing or re-architecting an existing system, you have the unique opportunity to prove that the new system is at least as good as the one it’s replacing. For rapidly growing companies like Pinterest, the necessity to re-architect or replace one component of our infrastructure happens relatively frequently. We rely heavily on logging, monitoring and unit tests to ensure we’re creating quality code. However, we also run A/B experiments whenever possible as a final step of validation to ensure there’s no unintended impact on Pinners. The way we run the experiment is pretty simple: half of Pinners are sent down the old code path and hit the old system and the other half use the new system. We then monitor the results to make sure there’s no impact across all our key metrics for Pinners in the treatment group. Here are the results of three such experiments.

2013: A new web framework

Our commitment to A/B testing infrastructural changes was forged in early 2013 when we rewrote our web framework. Our legacy code had grown increasingly unwieldy over time, and its functionality was beginning to diverge from that of our mobile apps because it ran through completely independent code paths. So we built a new web framework (code-named Denzel) that was modular and composable and consumed the same API as our mobile clients. At the same time we redesigned the look and feel of the website.

When it came time to launch, we debated extensively whether we should run an experiment at all, since we were fully organizationally committed to the change and hadn’t yet run many experiments on changes of this magnitude. But when we ran the experiment on a small fraction of our traffic, we discovered not only major bugs in some clients we hadn’t fully tested but also that some minor features we hadn’t ported over to the new framework were in fact driving significant user engagement. We reinstated these features and fixed the bugs before fully rolling out the new website, which gave Pinners a better experience and allowed us to understand our product better at the same time.

This first trial by fire helped us establish a broad culture of experimentation and data-driven decision-making, as well as learn to break down big changes into individually testable components.

2014: Pyapns

We rely on an open-source library called pyapns for sending push notifications via Apple’s servers. The project was written several years ago and wasn’t well maintained. Based on our data and what we’d heard from other companies, we had concerns about its reliability. We decided to test out using a different library called PyAPNs, which seemed better written and better maintained. We set up an A/B experiment, monitored the results and found that there was a 1 percent decrease in our visitors with PyAPNs. We did some digging and couldn’t determine the cause for the drop, so we eventually decided to roll back and stick with pyapns.

Figure 1: Experiment results for replacing pyapns

2015: User Service

We’ve slowly been moving towards a more service-oriented architecture. Recently we extracted a lot of our code for managing users and encapsulated it into our new UserService. We took an iterative approach to building the service, extracting one piece of functionality at a time. With such a major refactor of how we handle all user-related data, we wanted to ensure  nothing broke. We set up an experiment for each major piece of functionality that was extracted, for a total of three experiments. Each experiment completed successfully showing no drop in any metrics. The results have given us strong confidence that this new UserService is at parity with the previous code.

We’ve had a lot of success with A/B testing our infrastructure. It’s helped us identify when changes have caused a serious negative impact that we probably wouldn’t have noticed. When they go well, they also give us the confidence that a new system is performing as expected. If you’re not A/B testing your infrastructure changes, you really should be.

By: John Egan is a growth engineer and Andrea Burbank is a data scientist at Pinterest

Acknowledgements: Dan Feng, Josh Inkenbrandt, Nadine Harik, Vy Phan, John Egan and Andrea Burbank for helping run the experiments covered in this post.

Originally published on the Pinterest Engineering Blog

Long-term Impact of Badging

When it comes to growth, one potential pitfall is over optimizing for short-term wins. Growth teams operate at a pretty fast pace, and our team is no exception. We’re always running dozens of experiments at any given time, and once we find something that works, we ship it and move on to the next experiment. However, sometimes it’s important to take a step back and validate that a new tweak or feature really delivers long-term sustainable growth and isn’t just a short-term win that users will get tired of after prolonged exposure. In this post I’ll cover how we optimize for long-term sustainable growth.

Last year, we started including a badge number with all our push notifications. For many people, when an app has a badge number on it, the impulse to open and clear it is irresistible.

Figure 1: Example of badging on the Pinterest iOS app

When we launched badges, we ran an A/B experiment, as we do with any change, and the initial results were fantastic. Badging showed a 7 percent lift in daily active users (DAUs) and a significant lift in other key engagement metrics such as Pin close-ups, repins and Pin click-throughs. With such fantastic results, we quickly shipped the experiment. However, we had a nagging question about the long-term effectiveness of badging. Is badging effective long-term, or does user fatigue eventually set in and make users immune to it?

 To answer this question we created a 1 percent holdout group. A holdout group is an A/B experiment where you ship a feature to 99 percent of Pinners (users) and keep 1 percent  from seeing the feature in order to measure the long-term impact. We will typically run a holdout group whenever we have questions about the long-term impact or effectiveness of a particular feature.

We ran a holdout experiment for a little over one year. What we found was that the initial lift of 7 percent in DAUs settled on a long-term baseline of a 2.5 percent lift in DAUs after a couple months (see mobile views in Figure 2). Then last fall we launched a new feature, Pinterest News, a digest of recent activity of the Pinners you follow. As part of News, we would also badge Pinners when there were new News items. As a result, News helped increase the long-term lift of badging from 2.5 percent to 4 percent.

Figure 2: Lift of the badging group over the holdout group for key engagement metrics

We also found that badging was effective at increasing engagement levels. We classify Pinners into core, casual, marginal, etc., and we found badging had a statistically significant impact on attracting those who would have fallen in the marginal or dormant bucket to instead become core or casual users. This finding was compelling since it proved that badging is effective at improving long-term retention.

Figure 3: The badging group drove more Pinners into higher engagement buckets when compared to the holdout group

Holdout groups have been an effective way for us to ensure we’re building for long-term growth. We also have hold out groups for features like ads, user education, etc. In general, holdout groups should be used anytime there is a question about the long-term impact of a feature. In the case of badging, it allowed us to understand how Pinners responded to badging over a prolonged period of time, which will help inform our notification strategy going forward.

This article was also published to the Pinterest Engineering Blog

Growth Hacker TV Episode

Here is my Growth Hacker TV episode. Be sure to view their other great interviews at http://www.growthhacker.tv


Data Driven Growth

Here is the video of a talk I gave recently at the Weapons of Mass Distribution conference run by 500 Startups. I spoke about how you can use data to help drive your growth strategy and figure out which areas of growth you should be focusing on.

Hiring Growth Engineers Is Not Impossible

I read a post recently about how it is impossible to hire growth engineers. I’ve been lucky enough to have the opportunity to work on the growth engineering teams at two successful companies over the past few years. In that time I’ve learned that just like with any engineering role, hiring growth engineers is hard, but it is not impossible. Here’s how:

1. Cultivate Growth Engineers

Whenever any engineer joins a new organization, there is a learning curve and they need to learn & adjust to numerous things. They need to learn the company’s culture, their technology stack, their coding practices, etc. Growth engineering is a relatively new discipline, so the best strategy is to look for engineers that have the qualities that makeup a good growth engineer and help cultivate them into being a successful growth engineer. Teach them the importance of quickly shipping features and learning fast, teach them what MVP means, teach them to analyze the results of A/B experiments.

2. Look for Full Stack Engineers

Growth teams by nature touch the entire technology stack of a product: backend infrastructure, web frontend, Android, and iPhone. However, growth teams are also often extremely resource constrained and have a keen sense of urgency. Experienced engineers that can develop on multiple parts of the stack and quickly learn the parts they don’t know when necessary are critical to a growth team’s success. They give the growth team a huge amount of flexibility to move engineers around when projects/priorities change and also help keep the growth team from getting blocked.

 3. Product Sense 

Engineers believe it or not can have a keen product sense. This is evidenced by the numerous Product Managers that I’ve met over the years that had a CS background. Finding engineers who have a passion for being involved in product is important because they will be the engineers that are engaged by the mission and challenges of growth. They will also be the engineers passionate enough to really think about the user’s experience and mindset as they build product features to create a polished experience on their first try.

 4. Find Creative Problem Solvers

Often times in growth you have to find creative solutions around certain technical limitations. In my mind this is really where the term “growth hacker” comes into play. Whether it is finding a way to track invites on iOS (which has no attribution tracking for their app store) or figuring out how to generate recommendations for people to send invites to from a user’s contact list, creativity plays a large role in the success or failure of growth features. There is no substitute for creative problem solving, you either have it or you don’t.

Hiring growth engineers can be hard, but it is not impossible. Many of the best growth engineers I know came from standard engineering backgrounds, but they had the right mix of skills and were able to grow into the role.  If think you have these skills and are interested in learning how grow a company to millions of users, Pinterest is hiring growth engineers.

The Growth Hacker’s Guide to Push Notifications

Push notifications (aka PNSes) can be a powerful re-engagement tool for mobile apps to communicate to their users. In this post, I’ll cover everything a growth hacker needs to know about push notifications on iOS and Android.


One of the biggest differences between the two platforms is the permissions model for push notifications. On iOS, push notifications are opt-in. Users see the all too familiar dialog “AppName would you like to send you push notifications.” On Android however, push notifications are opt-out, users have to explicitly go into their settings and turn them off. This difference in permission model ends up having big implications for the effectiveness of PNSes on the different platforms. On Android generally 95%+ of users receive PNSes, while on iOS it us usually less than 50%.


Badging is the term used for the little red circle displayed on app icons in iOS. Badging has traditionally been one of the most powerful ways to use push notifications to re-engage users. One reason for its effectiveness is that many users can’t stand having unread notifications and will open the app to clear the badge number, giving the app a chance to re-engage the user. However, the primary reason badging was so effective is that prior to iOS7, apps that asked for the push notification permission would get a push token even if the user denied the push notification permission. The app could then use this push token to display a badge number and try to re-engage the 50%+ of iOS users that do not receive push notifications. Unfortunately, starting in iOS7 this is no longer the case. The good news however, is you can still continue to badge every user that first downloaded the app on iOS6. Another thing worth noting is that badging is no longer exclusive to iOS. Several Android manufacturers have started to add proprietary APIs to support badging on Android.

badge icon

Local Notifications

Local notifications are a way of getting around the permissions issue on iOS. Local notifications look and act like PNSes, but as the name implies, are scheduled by the app to appear at a certain time and date, rather than being pushed to the app at any time by the server. The major advantage of local notifications is that they are not tied to the push notification permission on iOS. This means even if the user does not give the app permission to show push notifications, the app can still show local notifications.
Edit (June 2, 2014): Starting in iOS8 local notifications are no longer a work-around for notification permissions

Geofenced Notifications

Geofenced notifications are when an app monitors the user’s location and shows the user a notification when they are near a place where the app would be useful. A classic example is shopping apps like Shopkick, or RetailMeNot show notifications when users are at the mall.  It can be tricky to get geofencing right; I’ve written before about some of the common pitfalls of geofencing. However, geofencing is still a very powerful tool, especially for location-based apps. Notifying the user when they are at a place where the app has content not only reminds the user to use the app in a situation where the app can add value, but also allows the app to deliver hyper relevant notifications.

Picture PNSes

Standard push notifications can be difficult to work with since all you have to grab the user’s attention is about 100 characters of text. One thing unique to Android is the ability to include an image that is displayed alongside the push notification. Android Picture Notifications make PNSes richer and more importantly, more engaging.

Emoji characters

While iOS doesn’t support picture notifications, it does have some support to spice up the text in push notifications. On iOS it is possible to include emoji characters in the message to try and make the message stand out and grab the user’s attention.

emoji in push notification

If you think there is something about push notifications that I missed, feel free to contact me at [email protected]

Why You Should Be Using ROI To Run Your Growth Team

The very first growth team I was on was run very democratically. Every month we would brainstorm new projects, everyone would vote on the ones they found most promising, and then we would execute on everyone’s top picks. After a few months, the team had delivered several projects that had beaten control, but we still weren’t seeing any visible impact on our bottom line metrics. To find out why, we took another pass through our experiment data and we realized that several of the experiments we had run never had a chance at impacting bottom line numbers. The reason was several of the features we developed were so far down the funnel, they would never reach enough users to move our numbers.

An example of such a project was our Facebook app. The company was a mobile only company, so when people came across us on the web we had to get them to make the jump to mobile and download our app. In order to better convert traffic from Facebook Timeline posts, we built an interactive Facebook app that would give users an idea of what the mobile app was about rather than sending them to a static landing page that instructed them to download the app. The Facebook app performed well, it increased our conversion rate from click to signup from 3% to 6.5%. The problem was, we only received 300 clicks/day from Facebook Timeline posts. This project earned us a whopping 11 incremental users/day. Even if we were able to 20x our Facebook traffic, the project would not have had a meaningful impact.

We realized the problem was with how we were selecting projects. Often the projects people found more exciting, such as building new features, would win over more boring projects such as optimizing the copy on invite messages.  We resolved to fix the issue by evaluating all growth projects in terms of the ROI on our bottom line metric of retained MAUs (a user that is still a MAU more than 30 days after signup).

To illustrate how we used ROI, lets go through an example. Say you think you can increase SEO traffic by 20%. Currently you get 10MM hits from SEO a month, so a 20% increase will net an additional 2MM hits/month. Normally 5% of your SEO traffic signs up for your site, so you can expect to net an additional 100K signups. Typically 50% of signups from SEO convert to a retained MAU, so you can calculate that this project will likely net you an additional 50K retained-MAUs/month. Now lets say this project will require an investment of 20 man-hours to complete. Calculating the return/investment gives us 2,500 retained-MAUs/man-hour.

Picking projects through the lens of ROI helped the team become much more effective with our time and resources by not wasting efforts on projects that sounded good in theory, but in reality would have no impact. I now strongly advise any growth team I talk to, to use ROI if they aren’t already, and to first understand the impact their various project ideas will have, instead of executing on them blindly.