Growth Engineering Meetup (Airbnb, Pinterest, Twitter)

Pinterest recently hosted a SF Growth Engineering meetup at the Pinterest office to discuss how some of the most successful companies drive growth at scale. See the video below to hear what speakers from Pinterest, Airbnb and Twitter had to share.

Synopsis of the talks

Paid growth @ Airbnb

Ganesh Venkataraman discusses the various channels and techniques used towards paid marketing at Airbnb as well as the underlying challenges around data/machine learning and attribution.

Speaker: Venkataraman leads guest growth engineering at Airbnb. Previously he worked at LinkedIn where he led careers relevance (job search, recommendations and statistical insights on salary). He has over 20 publications/tutorials/talks in leading conferences including SIGKDD, SIGIR, CIKM, SODA etc. He also has filed 16 patents in areas related to search and machine learning.

Onboarding component framework @ Twitter

Michael Lin covers Twitter’s Onboarding Component Framework which allows backend systems to construct personalizable signup flows using generic client-side components. This talk addresses the infrastructure, experiments and learnings from building this system.

Speaker: Lin is an infrastructure engineer on the Onboarding Team at Twitter. During his four year tenure at Twitter, he has also worked on account security, anti-spam and ads infrastructure. Michael is a graduate of the University of Toronto.

Keeping your resurrected users out of the graveyard @ Pinterest

Pinterest has millions of dormant users that resurrect each week, but often times they churn back out. Allan Blair discusses how Pinterest has built a strategy to classify, measure impact and ultimately increase our resurrected user retention.

Speaker: Blair is a full-stack engineer on the Engagement Growth team at Pinterest. Previously he worked as an Investment Banker at J.P. Morgan on Wall Street. Allan is a graduate of Rensselaer Polytechnic Institute in upstate New York.


The SF Growth Engineering Meetup holds these talks like these about once a quarter. Feel free to j. Join the SF Growth Engineering Meetup Group to stay in the loop and find out about future events.

AMP In Email And What It Means For The Future of Email Marketing

It is not often that something new comes along in the email space, much less something that has the potential to change email marketing as we know it. In case you missed it this week,  Google announced plans for Gmail to support AMP (Accelerated Mobile Pages) across all Gmail clients. While it isn’t immediately available today, Google expects it to be available sometime later this year. For CMOs, Heads of Growth, or anyone interested in Growth or Email Marketing, here is a primer on what AMP in email could mean for you.

What is AMP in email?

For a couple years now, Google has been promoting AMP as a standard that websites can use to make fast and responsive mobile web pages. AMP is a variant of HTML that has a number of restrictions on things that can sometimes make websites slow. AMP also adds a number of new HTML elements for common interactive UI components.

For those who have been lucky enough to avoid ever having to write email HTML, I’ve heard it best described as writing HTML as if it was 1999. There are a huge number of different combinations of email clients, operating systems, email service providers, and browsers. There are a lot of quirks or bugs in different email rendering engines, which means you have to write HTML for the lowest common denominator. Slowly over the past decade, some of the worst clients have been aging out and there has been an increasing trend towards something called “Kinetic Email”. Kinetic Email, which has been fully supported on web for over a decade, takes advantage of some creative hacks of GIFs combined with some basic CSS animations to make emails a little bit more interactive.

AMP in email takes the trend of Kinetic Email and blows it out of the water by now allowing you to write fully functional (albeit simple) web applications in email. You can literally now deliver a simple interactive web application straight into people’s inboxes. To see an example, watch this video of what Pinterest was able to do with AMP in email.

 

What does this mean for Growth?

Growth people will need to be thoughtful with how they use this. Growth people often want to bring users back to the product since that is where you can make revenue (ad impressions, purchases, etc.) There are a few strategies you will need to think through for AMP in email:

1) Do you use AMP in email to improve the chances of someone coming back in the future? For Pinterest, activation is often about gathering signals from users, so getting them to give us signals about their interests can improve retention even if they stay within Gmail and don’t click through to the product.

2) Do you use AMP in email simply as a tool to boost your overall CTR rate by making the email more interactive and engaging?

3) If users can now engage in core features of the product in email (i.e. make bookings, purchases, etc.), do you start to think of email as another client in addition to Web, iOS, and Android? Do you start to count people who take actions through your AMP email as DAUs?

 

What are the challenges ahead?

1)  AMP in email is an open specification, but we will have to see if this remains a Gmail specific feature or if other industry players (ESPs, marketing tools, WSIWYG design tools, etc) begin to adopt it. If it does remain a Gmail only feature, Gmail has such a large market share that it will still be worth adopting for any company that relies on email for re-engagement. Development time is on-par with traditional email development, even though the functionality is much richer;  it shouldn’t need to be a huge investment for companies.

2) Security. Gmail has already put a lot of thought into security with regards to phishing attacks, etc. However, as companies start to adopt this and explore what is possible, you can be sure scammers will be quick to exploit any potential vulnerabilities in user behavior.

3) Consumer adoption. This opens up a whole new way of interacting with emails. While it may be easy for companies to experiment with interactive email, there will be a learning curve for consumers to really understand that they can now engage with emails beyond just clicking a CTA button. The impact that AMP in email can have will increase as more and more consumers start to comprehend that they can interact more deeply with email and start to become more comfortable with doing more complex interactions over time.

 

Conclusion

AMP in email is exciting and perhaps one of the biggest potential innovations in email in the past decade. TechCrunch took a surprisingly luddite view on why “AMP in Email is a Terrible Idea“. Like with anything new, there are challenges ahead, but this has the potential to make email a lot more functional and engaging communication channel. To get started with checking out what is possible with AMP in Email register for the developer preview.

 

Acknowledgements: Seth Weisfeld for co-authoring this post. Susan Su from Reforge for inspiring me to write this post.

All the Advice You’ve Read on Push Permission Prompts is Wrong

For the last few years, it has become a widely promoted industry best practice to have pre-prompts for iOS’s push notification permission. This advice first originated from this 2014 blog post by a small startup called Cluster, which got picked up by TechCrunch at the time. Since then, it seems like every mobile analytics company has written a blog post repeating this advice: Localytics, Braze (formerly AppBoy), Optimizely, CleverTap, UrbanAirship, and the list goes on. The only problem is that, at least when it comes to push notification permissions, this advice is flat out wrong.

For those who aren’t familiar with the problem, you only get one shot when you pop up the system permission dialog for the push notification permission on iOS. If the user declines to give you the push permission, you cannot send them a re-prompt ever again. The theory behind pre-prompts is that you use pre-prompts to hold off showing the system permission dialog until the user is primed to accept. While this all makes complete sense in theory, it falls apart in practice.

For the past 4 years, I’ve worked on the Emails and Notifications team at Pinterest. During that time, we’ve tried more than a half-dozen different attempts at using pre-prompts to improve our opt-ins on push notifications. We’ve not only experimented on iOS but have also experimented on the permission prompts for browser notifications as well (Chrome, Firefox, Safari, etc.) The attempts have ranged from very simple pre-prompts with multiple re-prompts to complex flows to try and prompt users at a time where they could understand contextually why they should turn on push notifications. At my previous company, Shopkick, we even tried more “growth hacky” type approaches like putting a pulsar underneath the “allow” button to not so subtly suggest which button they should click. Even though all these experiments took different approaches, different framing, and different designs, none of them were successful. Every single one of these experiments showed a drop in engagement metrics across the board when compared to just popping up the system prompt.

In spite of all these experiments, I thought maybe we were just getting something wrong. Maybe there was something we were missing? I went and talked to other people working on push notifications at scale to find out. I’ve met with people working on notifications at a social network company based in Menlo Park, a music streaming company, a professional social network, and many more. When I talked to people at these companies working on notifications at scale, they all told me the same thing. They had all tried pre-prompts (and often had tried several iterations), but they had never found anything that beat just popping up the system permission dialog.

Pre-prompts aren’t all bad. I’ve seen them work for other permissions like photos or microphone. However, when it comes to push notifications, users make up their mind based on the category your app is in and there is little you can do to sway it. So, don’t put users through the hassle of two prompts just to turn on notifications, or even worse, put them through a pre-prompt and multiple re-prompts when they don’t want notifications. Just pop up the system dialog and get it over with.

P.S. If you’ve had success at scale with pre-prompts for the push notification permission, reach out to me at [email protected]. I’d love to hear about your experience.

3 Steps to Cracking International Growth

Cracking international growth is critical for any product that aspires to grow to hundreds of millions or billions of users. One of the key challenges with growing internationally is you can’t rely on your own intuition and experiences the same way you can when trying to grow in your native country. When I joined Pinterest, it was predominantly a US based service and had only just begun the process of starting to grow internationally. Over the next few years however, the company figured out what it took to grow the product internationally and mid last year we crossed a major milestone when over 50% of our active users were based outside of the US. In this post, I’ll cover the three fundamental problems a service needs to solve to be successful growing internationally and how to solve them.

The first and most basic problem a company needs to solve is localizing the product. At the most basic level, this means setting up a system to translate every line of text in the product. What people new to growth sometimes fail to appreciate is how much work it takes to really do this well. The words used in the product are the words new users read to understand the value of the product and how to use it. If the descriptions in the product don’t make a lot of sense, users will have a hard time figuring out how to use it and will ultimately be more likely to churn. At Pinterest, we saw a powerful example of how significant of an effect localization can have on comprehension and growth when we changed the text on the “Pin it” button to instead say “Save.” In user research, we had found that the metaphor of pinning things to corkboards wasn’t as prevalent in some cultures and therefore the entire metaphor of Pinterest as a product didn’t make as much sense. After changing “Pin it” to “Save,” we saw significant increases in Growth metrics in non-English speaking countries. As a result, we’ve started to take a very metrics-oriented approach to localization by building Copytune, a framework that allows us to test copy and pick the best performing variant on a per language basis.

The second problem is getting product/market fit in international markets. Just like any new startup needs to reach product/market fit when are first starting out, you need to revisit if you have product/market fit in important international markets when you begin to expand globally. One great example of this was when Uber added a cash payment option in India because many Indians did not have bank accounts or credit cards. Another example was when Facebook added blood type as a profile field in Japan because the Japanese believe that blood type imbues a lot of traits on a person’s character and is an important part of modern Japanese culture. It could even be as simple as improving performance so the site doesn’t take 20 seconds to load. For Pinterest, we ran several user research studies in key markets when we began expanding internationally. We discovered that while the website was properly localized when an international user signed up, their homefeed would be full of American pins with English descriptions. For us, reaching product/market fit was about updating our algorithms that powered search, homefeed, etc. to ensure that we surfaced locally relevant content. In other words, when a French user searched for recipes, they should find suggestions for local French cuisine unless they explicitly tried searching for international cuisines.

The final problem is building out your playbook for how to unlock Growth in a new country. After a startup first starts to gain traction, they go on a discovery process to figure out which growth channels work for them (SEO? Emails? Notifications? Paid Ads?, etc). Nailing international growth requires figuring out how to scale your tried and true growth channels to international markets. A great example of this again comes from when Facebook expanded into Japan. Facebook relied heavily on viral invites, but Japanese users initially didn’t send many invites. After doing some user research, they discovered that Japanese people were much less likely to send an invite to their friends asking them to join the service because they were worried their friends would find it bothersome or rude. However, when Facebook changed the language from “invite your friends” to “announce you joined Facebook,” they saw a big uptick in invites. Japanese users did not perceive announcing their presence on the service to be rude in the same way they perceived asking a friend to join the service. Other companies have had success in Asia when they adapted their invite flow and added locally popular messaging platforms such as Line, WeChat, Kakao Talk, etc. as options in their flow. You ultimately want to build up a playbook for how to tackle analyzing and building up your growth channels that you can then repeat on a country by country basis as the company starts to expand.

International Growth is a tough problem, and I’m only able to scratch the surface in this post. When I joined Pinterest, I was eager to learn how to scale a company internationally since my previous Growth work at Shopkick had only been based in the US. What I’ve learned is that success growing internationally requires taking many of the lessons from scaling in your native market and then figuring out how to adapt them to different countries and cultures. It requires nailing comprehension, making sure you have product/market fit for each country, and figuring out how to adapt your growth channels for individual markets.

One Simple Logic Error That Can Undermine Your Growth Strategy

Survivorship bias is one of the easiest logical errors to make in Growth. If you’re not familiar with survivorship bias, it is best explained with a short example. During World War II, the US wanted to add a limited amount of armor to their long-range bombers in order to minimize their losses without compromising fuel efficiency or payload size. To do this, the military surveyed planes which had seen combat and decided to place the armor in the areas that were receiving the most damage. The military’s top brass was moving along with the plan until a statistician, Abraham Ward, came along and pointed out that this was actually the worst place to put the armor. He had the insight that all the surveyed planes had been able to withstand the damage in those areas and make it back to base. Abraham’s conclusion was that the armor should actually be placed in the areas where all of the surveyed planes received no damage, since the planes that were hit in those areas did not make it home.

In Growth, there are a few different ways that survivorship bias can manifest itself. In this post, I’ll cover three examples of how I’ve seen survivorship bias undermine people’s understanding of Growth and Growth strategies.

Why Conversion Rates Sometimes Decay

One of the ways survivorship bias can crop up is when conversion rates decay over time. Let’s say you launch a new signup prompt on a landing page or a new invite email that is particularly effective and significantly boosts your conversion rate. Immediately after shipping the experiment, everything looks great. However, when you come back 6 months later, you see that the conversion rate has steadily dropped back to your old baseline from before the experiment launch. What is happening here? Well, when you first launched the new signup prompt, it was something new that no one had seen before. Some percentage of people seeing the prompt for the first time decided to signup. Those people are now no longer part of the population seeing the prompt because they have accounts. After that, a portion of people seeing the prompt for the second time decides to signup. Again, they are now removed from the population. For companies growing at scale, eventually you’re left with a significant portion of the population who may have already seen that particular prompt 5 or 6 times and weren’t convinced during those first 5 times and will likely continue to not be convinced from seeing the exact same prompt. What this means for a Growth team is they need to budget time to revisit important conversion prompts with the expectation that as users get acclimated to the prompt, its effectiveness will start to decline. On a broader scale, it might also mean you need to change up your entire conversion strategy as you move further along the S-curve of adoption.

 

Existing Users Are Biased

Another way that survivorship bias can manifest itself is in your existing user base. Every active user you have today has figured out how to use the product and is getting enough value to continue to use it. Everyone else that didn’t get it has probably already churned out. When I was at Shopkick, the Growth team placed a big bet in building geofenced notifications to alert users to the number of points they could get when they were near one of our partner stores. The initial experiment results showed only a 2-3% lift in store visits, which was far below what we had been hoping for. We spent several weeks trying to debug the feature and trying to figure out why it wasn’t driving more visitation. After wasting several weeks, we were going to just chalk it up as a minor win and stop investing more in geofenced notifications. Just before we were about to give up however, we decided to segment the experiment analysis by new users vs. existing users and that is when we realized what was really going on. Amongst new users, we were seeing a 30% lift in store visits. In contrast, it was only about a 2% lift amongst existing users. This was surprising since we had not planned this as an activation experiment. Ultimately, this made a lot of sense. All our existing users had already figured out how to use the product and developed the habit of pulling it out when they were at the store. Everyone who hadn’t figured that out had already stopped using the app long ago. As a result of this insight, we decided to continue to invest more in geofenced notifications and grew it as a strategy for activating users.

 

Email Unsubscribes 

The last example I’m going to talk about where survivorship bias can mislead you is when it comes to email engagement. A big area in email/push notifications is determining the optimal frequency to notify people. People often look at the email engagement of their existing users to try and figure out the optimal frequency to send. However, this can be a heavily biased approach because your existing users who are still receiving emails have self-selected themselves into being ok with your current frequency. Most people who disliked the frequency have already unsubscribed. To truly start to figure out the optimal frequency to notify users, you need to test on a clean unbiased population: new users that just signed up. Only by experimenting on new users, who have never received emails/notifications from your service before, can you determine what the optimal re-engagement frequency for your product is.

 

 

Recognizing some of the subtle ways survivorship bias can undermine what you think you know can allow you to adapt your growth strategy and help you decipher counter-intuitive results.

3 Habits of a Highly Effective Growth Engineering Team

There are a lot of great articles about how to set up your Growth team for success [1] [2]. At product driven companies like Airbnb, Pinterest, Uber, Facebook, etc., the Growth team is made up in a significant part by engineers. Building a great Growth Engineering team is a crucial part of building a great Growth team. I’ve been working on or managing Growth Engineering teams for over 5 years and have seen Growth teams as small as 2 people to Growth orgs larger than 50 people. In this post, I’ll cover three keys to success to set your Growth Engineering org up for maximum impact.

 

Full Stack Staffing

One of the critical traits of an effective Growth team is execution velocity. Execution velocity is important because Growth requires a lot of rapid iteration to learn and explore new ideas. In order to maximize the execution velocity, you need to staff the team with all the engineers you need to work on all areas of growth. This means backend engineers, iOS engineers, Android engineers, machine learning engineers, etc. Depending on another team that has different priorities can potentially kill the Growth team’s velocity. Depending on another team often means you need to constantly negotiate to get things on their roadmap, and once you learn from an experiment and come up with follow up iterations; you need to go through the whole negotiation process again. This process can cripple Growth’s ability to quickly iterate and learn.

 

From my own personal experiences, I’ve seen what a difference a full stack team can make. At Pinterest we rely heavily on recommendations in emails. These recommendations used to be powered by another team that (rightly so) could never prioritize significant investment in the email pipelines over their other priorities. The growth team decided to hire our own machine-learning engineers to start working on the email recommendations ourselves and within six months the team was able to launch several improvements to email recommendations that were the team’s biggest growth drivers of the year.

 

Once you’ve gotten headcount, you then need to hire the engineers. I’ve written before about hiring Growth engineers. Since Growth is still relatively new, there aren’t a lot of engineers who have worked specifically on Growth before, so I look for engineers who would be a good fit for Growth. Some engineers are motivated by working on hard technical problems, some by crafting the pixel perfect product experience, but the engineers that work best on the Growth team are engineers who are motivated by making a business impact. The engineers motivated by business impact tend to be the engineers that are most engaged with their projects and will be the ones coming up with ideas to even further increase their projects impact (it also helps if they have some product intuition as well). Which leads into my next tenant, building a culture of ownership.

 

Culture Of Ownership

            Once you have the team staffed up, it is important to build the culture of the team. For Growth Engineering teams, building a strong culture of ownership around the projects that the engineers are working on leads to the best results in the long run. I’m a firm believer that an engineer who spends days or weeks working on a particular project is in a significantly better position to come up with ideas for how to further increase the impact of the project than the Product Manager or Engineering Manager who can only spend a fraction of their time thinking about the project.

 

On my team, I try to instill a sense of ownership where engineers act as a mini-PM for their projects. Engineers are responsible for their experiments beginning to end, starting from writing the doc about why we are running the experiment, implementing it, doing the final analysis and finally, making the recommendation to ship or not. They are also responsible for coming up with ideas to further increase the impact of their project beyond what was originally scoped. They are empowered to propose and run experiments on what they think will further increase its impact. I’ve seen time and again that setting the expectation that engineers are responsible for figuring out how to increase the impact of their projects and giving them the autonomy to try things out has increased the impact of the original project by 50-100% in some cases. Our team helps reinforce the culture of ownership and maximizing results by recognizing and celebrating wins. We make a point to call out when engineers went above and beyond on a project to make it successful.

 

Quality and Stability

Finally, there is a lot of pressure to move fast on the Growth team. However, when things break, that can be a problem if it hurts topline Growth or if it happens so often the Growth team is constantly in firefighting mode. The Growth team can be responsible for over a dozen major features including signups, logins, NUX (new user experiences), emails, push notifications, invites, app upsells and SEO. If you multiply these features by each platform, like mobile web, desktop web, iOS, and Android, the numbers of opportunities for something to break compounds, especially when you have hundreds of engineers committing new code every day. A minor outage on something like your signup flow can end up costing you hundreds of thousands of users when you’re growing at scale.

 

In addition to working on experiments to increase Growth, it is imperative that the Growth team works to build a culture around quality and stability. This means ensuring proper unit testing, alerting, and extensive monitoring are in place to protect existing growth. For instance, alerts should be set up to catch a sudden drop in signups, an on-call rotation should be then be able to quickly deal with it. The on-call person needs extensive stats and logs to be able to quickly pinpoint the issue and diagnose the fix. Finally, a post-mortem process should be in place to review how to prevent future outages.

 

Wrap Up

Over the years, I’ve seen first-hand how each of these attributes have helped amplify the impact of the Growth Engineering team. Full stack staffing allows the Growth team to execute with a high velocity. At Pinterest the Growth team has taken on areas not normally associated with Growth, such as content recommendation pipelines, in order to move faster. Building a culture of ownership where engineers own and are empowered on the experiments they work on and are actively encouraged to come up with new ideas to improve the impact of their projects. Finally, a focus on quality and stability helps ensure Growth isn’t constantly fighting fires and existing growth is protected.

The Wrong Way To Analyze Experiments

One of the biggest mistakes I see Growth teams make when it comes to analyzing experiments is focusing too much on percentage gains. I’ve seen it time and time again when Growth teams pat themselves on the back for 30% increase in this metric or 15% increase in that metric. In this post, I’ll dissect the problem around percentage gains and why Growth teams should avoid using them to when it comes to assessing the overall impact of an experiment.

 

The Problems

The first problem with reporting percentage gains is that every experiment has an audience bias. By audience bias, I’m referring to the characteristics of the users in the experiment can differ significantly in terms of demographics, engagement levels, etc. from one experiment to another. For instance, let’s say I run an experiment where I send users a notification when someone likes their post. That experiment is naturally going to have an audience biased heavily towards active users that are actively posting content. You need to be posting content in order to be eligible to receive a notification about someone liking your post. In that hypothetical experiment, I might see only a small increase (ex: 1%) in daily active users (DAUs) because many people in the audience might already be DAUs. However, if I run another experiment to send an email to re-engage dormant users, I might see a 150% increase in DAUs because so few of the users in the experiment would become a DAU organically.

The second issue with percentage gains is that it obscures the true business impact. Going back to the email example, while one experiment had a 1% gain and the other had a 150% gain, we actually have no way of telling which experiment was more impactful for the business. The critical piece of data that is missing is the baseline population that the percentage gain is increasing. The 1% gain could be on a population with 10 million DAUs, which means the experiment netted us 100,000 incremental DAUs. Conversely, the 150% gain might be on a population of 10 million dormant users of which 20,000 were coming back organically, which means that experiment netted us only 50,000 incremental DAUs.

 

An Example

When I first started on the Growth team at Shopkick, we made the mistake of looking exclusively at percentage gains. During the first several months, the Growth team shipped a lot of experiments that had great percentage gains (70% increase in signups coming from Facebook posts, 50% increase in-store visits, etc.) About 5 months after forming the Growth team, we realized we weren’t actually seeing our topline metrics move. Going back over the data, we realized that a lot of those big percentage gains we posted in earlier experiments weren’t actually impacting the bottom line. For instance, the experiment that delivered a 70% increase in signups in reality was only adding an incremental 20 signups a day. Looking at percentage gains had been blinding us and we needed to focus on absolute numbers if we wanted to gauge the true impact.

 

Not All Bad

So, are percentage gains completely worthless? While I just spent most of the post railing on percentage gains, there are cases when they can be helpful to look at. For instance, percentage gains can be used to gauge the effectiveness of the experiment by judging the percentage increases for one metric relative to another. For instance, if you send 10% more email to dormant users that resulted in 20% gain in DAUs, that’s a good indicator that users really like the email. The inverse correlation might indicate the email isn’t that great and you’re just making up for it in volume. Just don’t use those percentage gains when reporting the experiments impact.

 

Wrap Up

About a year and a half ago, at Pinterest we shifted to using absolute numbers on the Growth team when reporting results. It has helped us compare and measure the true business impact of experiments that range across many different surface areas of the product and many different segments of users. Absolute numbers have also helped us more concretely measure a team’s output, by summing up the absolute impact of all the experiments they shipped that quarter. In Growth, the number 1 priority is driving impact, and if you’re not using absolute numbers to measure your output, then you can’t be sure you’re doing that.

How Pinterest, Dropbox, and Yelp Drive Growth @ Scale

We recently held a meetup at Pinterest’s offices to discuss how some of the most successful companies drive growth at scale. We had a great turnout to hear speakers from Dropbox, Pinterest, and Yelp.

Video Link: https://youtu.be/-1ABjPs6da4

Our next meetup is September 28th, 2016 at Lyft’s HQ in San Francisco. We will have speakers from Lyft, Pinterest, Airbnb, and Yik Yak. Please RSVP here: http://www.meetup.com/Growth-Engineering/events/234131759/

 

Synopsis of the Talks

Using Machine Learning to Drive Growth at Pinterest

At Pinterest, we send different types of recommendation notifications to our users in order to help them discover new pins and boards. In this talk, we present how we use machine learning techniques to decide which notification types to send and how often based on the user’s historical engagement. Using these models we are able to deliver a personalized notification experience to our individual users.

Utku is a Software Engineer on the Growth team and focuses on engagement modeling. He previously worked at Linkedin as a Staff Software Engineer.

 

Building a Growth Team at Yelp

Over the last year and a half, Yelp’s Growth team was built from scratch.  We’ll cover building a growth team from scratch, how we approached it, what went well, and lessons learned.  We’ll walk through a few of our experiments as case studies, and provide some guidance on how we decided what to focus the Growth team on.

Alex is an Engineering Manager on the Growth team.  Previous to the Growth team, Alex was an iOS and web engineer at Yelp.

 

Growth Platform at Dropbox

In this talk, we cover the platform that we’re building at Dropbox to scale communication with our half a billion users across several different channels such as email, mobile/desktop notifications and in-app surfaces for onboarding, engagement and monetization.
Ruchi is an Engineering Manager on Growth Platform team at Dropbox. She has been at Dropbox for a little more than a year. She previously worked at Hearsay Social, an enterprise SaaS startup, as the tech lead for their social media based lead generation platform.

Aditi is a Product Manager on the Growth Platform team at Dropbox. She has been at Dropbox for 3 years and has worked on diverse product experiences across sharing, user identity, growth, and monetization.

How Pinterest increased MAUs with one simple trick

For many areas of growth, presenting your message with the right hook to pique a user’s interest and to get them to engage is critical. Copy is especially important in areas such as landing pages, email subject lines or blog post titles, where users make split second decisions on whether or not to engage with the content based on a short phrase. Companies like BuzzFeed have built multi-billion dollar businesses in part by getting this phrasing down to a science and doing it more effectively than their competitors.

At Pinterest, we knew copy testing could be impactful, but we weren’t regularly running copy experiments because they were tedious to setup and analyze in our existing systems. This made it difficult to do the type of iteration necessary to optimize a piece of copy. Last year, however, we built a framework called Copytune to help address these issues. The framework has helped us optimize copy across numerous languages and significantly boost MAUs (Monthly Active Users). In this post, we’ll cover how we built Copytune, the strategy we’ve found most effective for optimizing copy and some important lessons we learned along the way.

 

Building Copytune

When we decided to build Copytune, we had a few goals in mind:

1)   Optimize copy on a per-language basis by running an independent experiment for each language. What performs best in English won’t necessarily perform the best in German.

2)   Make copy experiments easy to set up, and eliminate the need to change code in order to setup an experiment.

3)   Have copy experiments auto-resolve themselves. When you’re running 30+ independent experiments (one experiment for each language), each with 15 different variants, it becomes too much analysis overhead to have a human go in and pick the winning variant for each language.

Copytune dashboard showing different winners among languages

To achieve these goals, we built a framework that mimicked the API for Tower, the translation library that every string passes through. We first had every string pass through Copytune, which would check the database to see if there was an experiment setup for that string. If so, it would return one of the variants. If the string was not in experiment, Copytune would then pass the string to Tower to get the correct translation of the string. A nightly job would then compile statistics on all the copy experiments and would automatically shut down experiments when there was enough data to declare the winner.

Copy optimization strategy

Testing copy requires an iterative process to achieve the best results. It’s almost impossible to identify the ‘best’ copy in one go, so we  took an incremental approach to discover it.

  1. Explore Phase: You can’t know for sure what will work, so we started by testing many variants that touch on very different themes, tones, etc. We typically brainstorm 15 – 20 different variants. For example:

    • The latest Pins in Home Decor
    • Come see the top Pins in Home Decor for 12/3/2015
    • We found a few Pins you might like
  1. Refine Phase: After the Explore Phase, we began  to see which tones and phrasing  were performing best. Then we could refine by testing different components of the winning variants of the Explore Phase.

Let’s say that in Explore Phase, the winner was “We found some {pin_keyword} and {pin_topic} Pins and boards for you!”. There are many possible optimizations we can test in this example.

Example of component variations

We can try adding “Hey Emma!” at the beginning to catch the Pinner’s attention. We can even test whether “Hi Emma!” or just “Emma!” is better than “Hey Emma!”. We can test some phrases like “we found” vs. “we picked.” We can test if “Pins and boards” is better than just having “Pins” or “Boards.” In this example there are at least 10 components we can test. We treat them as independent components and test each of them against the winner.

  1. Combine Phase: Let’s say “Hi Emma!”, “we picked” and “{pin_topic}” were winners in the Refine Phase. We can now test if the combination Refine Phase winners (a) performs better than the original winner (b)

1. “Hi Emma! We picked some {pin_topic1} and {pin_topic2} Pins and boards for you!”

2. “We found some {pin_keyword} and {pin_topic} Pins and boards for you!”

Note that it’s possible that some components are not independent, so we also tested other combinations that seem promising.

In one of our highest volume emails, the winning variant from the Explore Phase showed only a gain in one percent open rate. By the end of the whole iteration, optimizing the subject line on one email boosted it to an 11 percent gain, adding hundreds of thousands more active Pinners each week.

 

Lessons learned

Copytune has been in place for almost a year now, and we’ve learned some lessons along the way:

Defining Success: When we initially started testing email subject lines, we defined the success criteria as driving an email open. This seemed to be the most straightforward since the Pinner reads the email subject line and the next action is to either open it or don’t. What we found, however, was that defining success with metrics that were further downstream (i.e. clicking on the content in the email) was more effective. Some subject lines were great at getting opens, but there was a mismatch in user expectations based on the subject line and the actual content in the email so net net they were actually resulting in fewer clicks.

Picking Variants: The original vision for Copytune was to use a Multi Armed Bandit framework for picking variants and auto-resolving experiments. The difficulty we ran into was feature owners wanted to see how the experiment performed across a variety of metrics and to be able to report concrete MAU gains from the experiment. To accommodate these needs, we ultimately needed to integrate Copytune with our internal A/B testing framework.

 

Acknowledgements: Koichiro Narita for co-writing this post, helping develop Copytune, and running the subject line experiments covered in this post. Devin Finzer and Sangmin Shin for helping develop Copytune.

This post was originally published on the Pinterest Engineering Blog

4 Steps To Develop Your Push Notification Strategy

Startups often struggle with how to develop their push notification strategy. While email has been around for decades and is fairly mature, push has only been around a few years and people are still trying to get a handle on it. In this post, I’ll cover the basics of how to develop a messaging strategy that applies to both push and email and how to take advantage of some of the unique aspects of push.

Step 1: Define the product’s core value proposition

Push notifications should be an extension of the product’s core value proposition. I can’t emphasize this enough. One of the biggest mistakes I see startups make (and I’ve made myself) is that they send emails/notifications about things that are not strongly tied to the value proposition. The value proposition is the reason people engage with the product and is what sets your product apart. Push notifications should further that engagement and make it easier for them to derive that core value. For users who don’t yet “get” the product, push should help them understand the value. For users who do “get” the product, push notifications should help them engage further engage with it.

Step 2: Figure out what you can send that is tied to that core value proposition

Content generally falls into one of three broad buckets, each with their own pros and cons. The type of content you send depends both on what makes sense with the product and what your resource constraints are.

Marketing Driven: These are notification blasts sent out by the marketing team to most or all users. A lot of ecommerce and brick and mortar retailers fall into this bucket.

Pros:

  • Coverage: Can send to every single user
  • Minimal engineering effort required, which makes it great for early stage startups
Cons:

  • Content is not personalized, which leads to low engagement rates
  • Users have lower tolerance to these types of notifications, which means you have to use them sparingly to avoid high unsubscribe and app deletion rates

Transactional: These notifications are triggered by users’ actions on the service. They inform other users about those actions. Facebook and LinkedIn are great examples of this.

Pros:

  • Generally good engagement since content is relevant by virtue of the fact that the user has an direct connection to the action
  • Higher level of tolerance since users understand what is triggering the notifications
Cons:

  • Need to have enough engagement on the site, or be connected to enough users, to get the flywheel going

 

Content Driven: Content driven notifications connect users with relevant and interesting content. They generally use some amount of personalization to figure out which content to recommend.  Twitter for example will send emails/notifications to less engaged users about popular tweets they think the user will be interested in.

Pros:

  • Can get good engagement rates by sending highly personalized, relevant content.
  • Can get good coverage by sending trending and popular content to users for which you don’t have enough signal to personalize recommendations.
Cons:

  • Engagement rates get worse the less the user has engaged with the site
  • Expensive to build out recommendation algorithms from an engineering effort perspective

Step 3: Figure out your user segments

Once you’ve figured out what content you send, you then need to figure out who you want to send to. Not all notifications are good for all users. Notifications should be targeted based on where the user is in their lifecycle. A very simple but powerful segmentation is classifying users into new, engaged, and unengaged.

  • New Users – Send notifications that help reinforce the product’s value and help them figure out how to get more value out of the app.
  • Engaged Users –These users are already engaged and understand the product, so only send them the best, most useful notifications that help them engage even further.
  • Unengaged Users – Unengaged users are always the toughest nut to crack since they have already shown a bias towards not engaging with the product. The signals you have on them may or may not be accurate so sending a mix of personalized notifications and broader non-personalized notifications is necessary to try and re-engage them.

Step 4: Think about what makes push unique

Up to this point, everything we’ve talked about can apply just as much to email as it does to push. However, there are a few things that really differentiate push from email and may change your approach to push.

1)   Timeliness – Since most people have their phones on them at all times, push notifications allow you to reach users more immediately than email.

2)   Location Based – Both iOS and Android have good support for geofenced notifications that allow you to notify the user when they are near a certain latitude and longitude point.

3)   Badging – Badging is a way to give the user an indicator that there is something new in the app in a way that is less intrusive than sending an email or normal push, and still triggers a lot of engagement.

The final step is to ask yourself if there is any way these attributes naturally dovetail with your product’s value proposition. For example, geofencing notifications are a great fit for location based apps, but can feel out of place if location is not a core feature of the app.

Wrap Up

 As with anything Growth related, push notifications require a lot of trial and error, iteration, and experimentation. However, I’ve found thinking of push notifications as an extension of the app’s value proposition and then thinking through this framework has helped me a lot when crafting a push notification strategy.