John Egan

The very first growth team I was on was run very democratically. Every month we would brainstorm new projects, everyone would vote on the ones they found most promising, and then we would execute on everyone’s top picks. After a few months, the team had delivered several projects that had beaten control, but we still weren’t seeing any visible impact on our bottom line metrics. To find out why, we took another pass through our experiment data and we realized that several of the experiments we had run never had a chance at impacting bottom line numbers. The reason was several of the features we developed were so far down the funnel, they would never reach enough users to move our numbers.

An example of such a project was our Facebook app. The company was a mobile only company, so when people came across us on the web we had to get them to make the jump to mobile and download our app. In order to better convert traffic from Facebook Timeline posts, we built an interactive Facebook app that would give users an idea of what the mobile app was about rather than sending them to a static landing page that instructed them to download the app. The Facebook app performed well, it increased our conversion rate from click to signup from 3% to 6.5%. The problem was, we only received 300 clicks/day from Facebook Timeline posts. This project earned us a whopping 11 incremental users/day. Even if we were able to 20x our Facebook traffic, the project would not have had a meaningful impact.

We realized the problem was with how we were selecting projects. Often the projects people found more exciting, such as building new features, would win over more boring projects such as optimizing the copy on invite messages.  We resolved to fix the issue by evaluating all growth projects in terms of the ROI on our bottom line metric of retained MAUs (a user that is still a MAU more than 30 days after signup).

To illustrate how we used ROI, lets go through an example. Say you think you can increase SEO traffic by 20%. Currently you get 10MM hits from SEO a month, so a 20% increase will net an additional 2MM hits/month. Normally 5% of your SEO traffic signs up for your site, so you can expect to net an additional 100K signups. Typically 50% of signups from SEO convert to a retained MAU, so you can calculate that this project will likely net you an additional 50K retained-MAUs/month. Now lets say this project will require an investment of 20 man-hours to complete. Calculating the return/investment gives us 2,500 retained-MAUs/man-hour.

Picking projects through the lens of ROI helped the team become much more effective with our time and resources by not wasting efforts on projects that sounded good in theory, but in reality would have no impact. I now strongly advise any growth team I talk to, to use ROI if they aren’t already, and to first understand the impact their various project ideas will have, instead of executing on them blindly.