YC Partner and growth expert Gustaf Alstromer talks about how to measure product market fit, how to decide on a growth channel, metrics that lie about PMF and other mechanics of growth for startups.
Watch this if:
Gustaf: My name is Gustaf. I'm going to give a talk on Growth for Startups. This is going to be for some of you guys not super relevant right now, because you might not launched. And thinking too much about growth when you haven't launched isn't that relevant. But for those of you that have launched, this is hopefully going to be a good talk. So, I'm going to cover three different things today. First, I'm going to talk about product-market fit and retention. The reason that that relates to growth so much is because working on growth before you have product-market fit and good retention is not a good idea.
The second thing I'm gonna talk about is growth, channels, and tactics. These things definitely apply after you have launched, and often after you have a good product-market fit. You've found something that people really want, and then you want to scale it up to the larger world. And lastly, I want to talk about how you make decisions. When you have several people on your team, you want to start redoing things and you're not really sure exactly if you're making the right decisions or not. And these are all the things that apply when you're a little bit bigger.
So, my background, I learned most of these things I'm gonna talk about Airbnb. I worked on the growth team for almost 5 years, from where we were 2 people until we were over 100 people on that team. This is the team back in 2015. Most of the lessons I'll talk to you today are things I learned there. Most of you are going to be somewhere on this line. Most startups don't have product-market fit. Founders tell themselves that they do, and they try to convince themselves that this is working. But the truth is, for most companies, it's not working. So that means you're going to be somewhere on this line.
People also have this idea that if I launch my product, it will work. Somehow, it's going to work if I just tell the world that I built ...this is now there. Now, unfortunately, that's not the case. The world is a really busy place, and there isn't really lots of people waiting for you to launch your product. They're not standing there and they're not going to try it the moment you launch it. That is unfortunately not the truth. And for many people who have never thought of these questions before, of, "How do I reach the world?" this actually comes as a surprise.
People have been used to working in big companies where this is not a problem. People have been used to going to school or other areas where this is just not a problem. In this case, when you launch a startup, it's all down to you, and it is going to be a problem in the very early days. There's a great article that I recommend for you guys to read on this. This is called "Doing Things That Don't Scale" by Paul Graham. He wrote that six years ago. It is about the early days of the Airbnb story. And the thing that's really important about this, is as a founder, you need to keep two different skill sets in mind as your company grow.
In the beginning of your company, you're going to do a lot of things that don't feel right. They don't feel natural to do, because it's not the kind of thing that you learn in your previous jobs or in school. It's they're so, like, the most kind of physical or real things that you have to do that you aren't going to be relevant later on. But later on, as your company grows bigger, you're going to be doing a lot of things that are things that relate directly to software, are things that scale to company. So, these are two skill sets that you have to keep in mind at the same time. In YC, we have this thing where we tell companies that, "You just launched. You gotta do things that don't scale." And we got lots of these MBAs that went to school and said, "Well, this idea does not scale, standing outside of this store or standing in this elevator to sell people something, that certainly doesn't scale."
Correct. That does not scale. But that is where everyone needs to start. And if you went to school and you learned that you should only work on things that really scale, you're going to have to unlearn that skill. Because when you start your company, the most important thing is going to do things that don't scale. So if you get comfortable with that idea. This is the early days of Airbnb. So, this is sometimes in 2009. They were just a few people. The article I mentioned earlier, "Doing Things That Don't Scale," tells a story of the first year or two of Airbnb. When the founders came to YC, they had spent almost a year trying to get Airbnb off the ground. It didn't really work.
This was the first version of the Airbnb website, airbedandbreakfast.com. In fact, the website itself didn't really speak to what the company does. It was started as a website to offer air mattresses to people that visited design conferences, and they had to navigate their way to find the place to where Airbnb is today. When Airbnb joined YC, the first question they got from Paul Graham was, "Who are your users?" And at the time, the site looked something like this. You click on a listing, and you had three different pieces of information. You had a photo of the host, you have one photo, in this case, of the building, from the outside, and then you have one map of where that place was. Now, at the time, the only comparison to what a site like this would look like would be Craigslist.
So, Craigslist wasn't a lot better than this. So at least it met that criteria, but it wasn't something that would make Airbnb take off. They didn't really have, in the product, what would make Airbnb take off. The things that were missing is, is this a good listing? How does this listing actually look like? Can I trust the host? Lots of things that were missing in that early product. And how do you learn that? The way they learned that is they went and talked to their host. On their first week in YC, Paul Graham told the founders of Airbnb, "You guys go meet your host. Where are your host?" "Most of our hosts are in New York. We don't have that many, but most of them are in New York."
So, they flew to New York, undercover, not on the cover. They claimed to be hired photographers for Airbnb, Air Bed and Breakfast. So, when they met with all the hosts, they said, "Wanna come by your home and take photos." They didn't say that they were the founders, because that made the company sound much smaller. They came and met with a host, and while one of the founders were taking the photos of the listings, to make this look a lot better, the other founder sat down with the host and asked them questions about how, what are the challenges you're having with the product? Like, what are the things that are not working? Can you show me how you use the product?
And by doing that, they got for the first time to meet the people that were their customers, which they really haven't done before. And they got to see how they use their products. That's doing things that don't scale. And that is nothing that scales. You can't go and fly to meet every single one of your customers. But when you start doing that, you will learn things that you can't learn sitting in front a computer. So, they learned that this payouts thing didn't work, or there was a big UI bug on this page, or didn't work on Internet Explorer. Well, all these things that you can't learn sitting in front of your computer.
They went back to San Francisco, back to Y Combinator, and they sent an email the morning after and they said, "Here are all the photos we took of your house. They're now up on airbedandbreakfast.com. And by the way, we fixed half of the bugs that you emailed us about," or, "We fixed the bug that you told us about yesterday." That made the host love them. And those hosts became the reason that Airbnb eventually took off. Like, doing things that don't scale, fixing the product, making the product work for the early hosts, which became the backbones of the early days of Airbnb. So, the lesson here is that founders are the ones who make startups take off.
The founders, you guys, are the ones that make the startups take off. You're going to have to do unconventional things, you're gonna have to do things that don't feel right, certainly gonna do things that you didn't learn in business school, and you're just gonna do the things that are needed. And this is basically what the YC batch is about. When someone joins YC, we're going to be like, "You're gonna launch," because that's the most important thing you could do right now. But once you've launched it's like, "How do I get users?" Like, you gotta figure out how to do it. And it's different for every company. For many of the company, that means sales.
For other companies, that means doing things that don't scale. Typically, people start with their friends, and then friends of friends. And then hopefully, you get one step further, the people that are now not your friends of friends, and they're gonna give you a true opinion about your company. Those are the people that you're gonna have to reach early on. It doesn't really start with, like, "I launched my website, and I put up Google ads," or, "I launched my website, and somehow, it's being discovered." That's not how companies get started. That's how they end up much later, but that's not how they get started. There's only one way to grow when you're really small. And that is doing things that don't scale.
All right, next topic. I'm going to talk about product-market fit. This is a terminology that probably most of you have heard of. This is a thing that's been hard to measure, or hard for people to say, "Do I have product-market fit or not?" A lot of people like to tell themselves that they do have product-market fit. It's this thing that we throw around as a way to say, "My product is great, so now I have product-market fit." I would argue that there's some ways that you can measure product-market fit, and there are many ways that you can't, so let's talk about the one thing that I think is the best way to do that. I think that the best way to figure out if product-market fit, is to use data, unbiased data, to understand if you basically have made something that people want.
The two ways that I do that, or that I start that, the first way is to try to figure out what is the metric, the data point, that represents the value of your company? That's the first thing I do. The second thing I do is try to figure out, how often should I really be doing that? Great example might be a startup school. The metric here is, like, are people showing up to that video talks and startup school? How often is that? It's every week. All right, that's pretty easy. But most companies can be defined this way. Let's give some example. So, Airbnb. What is the metric represents the value? Well, it's the bookings and the stays. It's not the searches. Search is not that...it's going to be the bookings and the stay, when I travel Airbnb, I've experienced the value.
So now, we know what it is about. How often do people do this? Well, travel is actually mostly an annual thing. You don't really travel every month. Most people don't do that. So, when we were measuring retention at Airbnb, we were looking at annual. Let's look at Instagram. What's the expected use case of Instagram? It's basically just coming back to Instagram. Most people are not expected to post photos every day. It is just going to be coming back to Instagram and viewing photos. That's what most people do. And that's fine. That's actually what they want. They want some people to post photos sometimes, but most the time just coming back is good enough. How often? Probably every day.
Let's think of a B2B company, Gusto. So, for Gusto, the most valuable thing that they do for their customers is they run payroll. And they pay out money to the employees of Gusto's customers. So, how often do you run payroll? Well, it depends, probably every, biweekly, or monthly. And by measuring these two things, how many people am I running payroll, and are they continually running payroll with me? That's probably the best way to figure out if people enjoy using Gusto, if they're gonna switch to some other payroll provider.
And finally, Lyft. You might want to think it's rides here, that rides is the best metric here. It's actually riders, like, the people that are taking the rides are the ones that matter, because it's the individuals that we want to measure here. And we not necessarily want to measure the action that they take. And that's probably weekly or monthly. So now, we have these two metrics, we have a bunch of examples of those companies. Let's put on my graph. One piece on the graph is going to be the metric, and the other one is going to be the time window.
So, every single time window, we can put some percentage of those people on the graph. So, let's give an example. On week zero, in the case of Lyft, you had hundred percent drivers. So, what do I mean by that? I basically mean that if I had, like, say, I had 10 riders this week that rode with Lyft, they would be calculated on the week zero. Now, how many of the riders that I had last week are now traveling with Lyft this week? That is your week one number, and the week two number, and the week three number. Now, why is this important? Because we're trying to measure repeat usage.
Repeat usage is the best, most unbiased way to figure out if someone is liking your product. It's more true than what they tell you. They might tell you things, but what they do is going to be the most important thing. So, most companies can be defined this way. Even if you have a B2B contract company that do annual contracts, measuring, say, what do people do with my product, could be a really good way on, like, this regular basis. So, even if I'll pay for Gusto on an annual basis, which they don't do, measuring the activity using Gusto on a regular basis, let's say biweekly or monthly, is the way to figure out if people are actually using the product.
So, most of the ideas, even your B2B, or consumers, could be plotted on this line. Now, why is this important? Well, if you're ever going to raise money, this is a graph that investors are going to ask for. Like, how much retention do you have? Like, are people actually repetitively using your product? Those are the things they're really curious about, because they know there are other metrics that you might have that don't matter. This is a sign of a bad product. Basically, every single week after I started using this product, fewer and fewer and fewer people continue to come back to use the product. So, this graph can be plotted and basically show that this wasn't a good product.
This, however, is a good product. Every week, it eventually flattens out, and the people that stop...the product stops, stop using the product. And eventually here, at week 8, 9, 10, we have a flat line of people that continue to use the product every single week. That means that they are retained. You have product-market fit for those users for this product. So, I'm not going to ask you these questions, but here are two examples of two companies that I would argue have product-market fit. The first one here has 30% after two months, and 21% after 20 months. This is pretty good.
So you kept a fifth of your users 21 months later, or 20 months later. It's DoorDash. DoorDash have a monthly retention of 20% two years later, a year and a half later. Here's another company, more like a B2B company. So, 80% retention after one month, and then 30% after 60 months. This is really good. This is a really good product, very sticky. People like this product and they don't stop using it. It's GitHub. So, retention is the best way to measure product-market fit. Let's talk about specific thing you might want to...that some people think is a better way. I'll argue that they're not.
So, here's some worse ways to measure product-market fits. Net Promoter Score. Why is it not good? Well, you can just google the best products and best companies in the world. They all have bad net promoter score. Like the iPhone, Apple, all them have bad net promoter scores. Like, it doesn't necessarily correlate with good products, correlates with perceptions of companies. Surveys. The problem with surveys is they are going to be biased. So, if you ask your users, you're going to have some level of bias. There are good ways to use surveys to improve your products, but it's not going to be the best way to figure out this metric.
There is one cool question you can ask a user, which is, "How would you feel if you can no longer use this product?" Sometimes this works. It can give you an idea. But I wouldn't do it instead of retention. I'll just always try to find a way to measure retention. All right. So, what are some bad metrics for product-market fit? These are not the kind of things you want to throw around is, like, evidence for your product is working. Registered users, really bad. Does not say anything about repeat usage, or if they liked you or not. Visitors, also bad. Does not say anything about whether your product is going to be valuable. Conversion rate. "We have this conversion rate of visitors to something else."
Well, that doesn't really say much, because you don't know what people you're converting. You don't know who they are. So, this is not saying much about product-market fit either. And finally, something that should be a paid product, and you're giving it away for free, is not a good sign of product-market fit. Like, you want to figure out if people are willing to pay for it. Because price, if someone says, "I love this if it's free, but if it costs money, I'm not gonna use it," that's pretty bad. Like, it's not gonna work out for you. So, you want to make sure that the people that are doing something like this on this graph, if it's expected you pay for a product, they should be paying for the product.
All right, next section. Let's talk about growth channels and tactics. This section really applies if you have product-market fit. If most of the people that come to your product go down the drain right away and they never come back, this section doesn't matter. Like, why would you work on trying to get more people to a product that no one is using your product anyway? If most people are just churning, and, like, they try it once and then they don't come back, like, don't work at this stuff. Wait with this stuff until you have some people that care about your product, and you can try to use some of these channels to reach those people specifically.
There's really two ways that you can grow at scale. So, when I looked at that team, that sort of photo of the Airbnb team, they worked on two things. They either worked on what I call product growth, or conversion rate optimization. What this means is you have typically engineers, designers, data scientists, product managers, working on improving specific parts of your product to get more people through that funnel. So, good example, I'm gonna give you some example in a second. But that's basically what define as the first section. Most of those people in that photo, or in this category, they were engineers, designers, product managers, and data scientists.
The second group is what I call growth channels. Growth channels is basically platforms in the world that people tend to discover products on. Let me give you some specific examples. Google. It's huge platform for new products to be discovered. Anything that you want to use, that is a rare behavior in your life, Google, that's what you do. Insurance, don't forgot the insurance. Google. Want to find a doctor, Google. Everything you do rarely, it's going to be Google. Which means lots of products have been discovered on Google. And growth channels like Google is an extremely important one for many companies.
Another one might be Facebook, Instagram. Advertising on Facebook and Instagram is critical to companies' growth these days. What I mean by growth channels, that means basically, other platforms but your website, or your app. So, let's talk about conversion rate optimization. What does it mean? Every single step of your product experience is a funnel that, like the retention curve, can be measured. You can have a metric, and I think Ilya talked about this earlier, startup school talk, when they build funnels. If you put a metric on every single page in your product, you will know what percent of people that make it from the first page, say the home page, to the booking page.
In the case of Airbnb, we call the homepage P1, the search results page P2, and the listing page P3, and then the booking page was P4. Four pages. That was the entire website. Now, what's the funnel? What percent of people make it from P1 to P4? What percent? Not that many. One percent, two percent. Most people don't make it that far. Your job is to figure out how many people make it that far? Why are they dropping off? What can I do to increase the number? That's basically multiple teams, or multiple people at startups, that work on those things. Every single step in that funnel is going to have some kind of drop off for some reason.
They might be that the content on the page is not suited for them. I land on Airbnb, all the content speaks to millennials. I have a family. It's not good content. I land on some other website, and the content doesn't speak to me because I'm not the right customer. That's one example of a drop off that you can fix with changing the content. Another one might be, I'll land on the website. It doesn't work, because Internet Explorer is not optimized for that. It's not optimized for that. You're gonna drop off. So you gotta fix that too. The lots of different reasons people want to drop off. Here's the specific things that people tend to work on when they work on conversion rate optimization.
Internationalization. If your website or your product is international, translating it, the product, is really a good idea. We saw that at Airbnb, I've seen that in Facebook, I've seen that in many other companies, where translation is really, really important. Authentication. Most products have some flow where you're signing up. Now, that flow, probably your products too have some kind of authentication flow, that flow is very critical, and the users are kind of vulnerable in that case, because they don't really have time for too much friction. So if it's not working perfect, they might just go to the next website. So, make sure the authentication flow works really well. Look at the best websites in the world. Look at Pinterest, look at Airbnb, look at some of those sites. They have teams optimizing these flows, the authentication flow. Copy what they do. They've probably figured it out. They've spent a lot of time optimizing.
Onboarding. This is a huge effort, specifically for products that need a lot of involvement from the users to be able to become active users. So, there are a lot of questions you might want to ask early on in a new product. The more you can onboard users by asking them questions that make the experience better, the more active and the more retained they will be. So, onboarding, there's lots of things you can do.
And finally, purchase conversion. When you're about to purchase, the lot of things around urgency and scarcity and just user flow and UI, all of these things matters. And that's another great example of conversion rate optimization.
So, let's talk about growth channels. So, again, don't get here until you have some good sense of that this is something people want. First one, like I said earlier, if this is a rare behavior, most new ideas are rare behaviors, either because they don't exist yet, or because they're not something you do every day. We tend to go to Google to learn about things that we don't do very often. So that's why, if that is the kind of product that you have, being on Google is going to be really important.
You can either be on Google through paid marketing, through SEM, or through SEO. I'll talk about in a second. Second, does your product already...the people already share your product through word of mouth? So, some products are viral in its nature because it sounds really exciting to talk about. Lyft and Uber and Airbnb are examples of those. If that's the case, you want to make sure you focus on virality and referrals. What does that mean? Means you're building into your product a flow that friends can tell other friends about the product. Referrals is a way that you can do that by giving some kind of financial incentive.
Does the product get better if you have more users? Well, this is true for marketplaces. But it's specifically true for anything that's social. So, if you think of a LinkedIn or a Facebook, then having more people on the product is going to make it better. So it's going to be really important for you to get more people, and those people on your site's going to be the ones doing it. So you want to figure out a good viral loop. So, when you sign up to LinkedIn, the first thing they ask you is to invite more people. That's because your experience get better when there are more people on LinkedIn. Now, many products do work this way, and this can be perfected. And the ones who really succeed in the world of a social product are the ones that really nail this down. They figure how to do this really well.
Many people that make social apps underestimate how important it's going to be to get your friends on that product. If you can make a list of all their customers, even if that list is 100,000 or 500,000, as long as it's not mainstream enough that it'll be in the tens of millions, you're probably gonna do sales. You'll make that list, and you start counting those people. Why make it any more complicated? Why go out and reach the world if there are only a few people that you really want to reach? So, most companies in YC these days, I ask them this question, "Can you make a list of your customers?" "Yeah." "Right, make that list. Start listing them out. Who are the people? Decision makers in those companies you're trying to sell to. These people. Make the list, email addresses, phone numbers, trying to figure out how to reach them."
But start by making the list. Don't make it complicated by going out in a world where most people aren't going to be relevant for your product. And finally, this is a channel that nowadays is bigger than it ever has been, and more important than it ever has been, which is, if you look at how the entire world of startups has changed the last 10 years, more and more of them are churning more money, and therefore getting what's called a high LTV, high lifetime value.
By getting high lifetime value, you're enabling the ability to buy paid advertising. If you don't have people paying for your product, where you're making money from your product, you should not be spending time on online marketing. Now, the truth is that most companies these days are charging for the products, and they are making money from the product, and therefore they spend money on online marketing. If that's true for you, this can mean extremely powerful channel. The biggest mistakes founders make is to start working on online marketing when they don't have people paying for the product.
Here's an insight that you probably didn't think of. Most really big companies didn't use all of those channels. They use one or two channels. So, think of a TripAdvisor. How is TripAdvisor big? SEO. You guys type in something on Google, you land on TripAdvisor, and that's how you found this website. Most companies have a setup where there's gonna be one or two channels that really matters. If you think of Pinterest, SEO is the real way how Pinterest is growing. You type something on Google, there already exists a Pinterest board for that, and you land on that Pinterest board. That's how they acquire new users.
All right, I'm gonna give some specific tactical advice on some of these channels. The first one I talked about is referrals and virality. So, referrals is word of mouth. If word of mouth is a strong driver of your product, then referrals is gonna be one way that you can amplify that word of mouth. How do I define referrals? A financial incentive to tell your friends about the product. That's my definition. This is the Airbnb referral product. You gave someone $40 to sign up to Airbnb. And when they do, I get $20. Pretty simple concept. We have that on the website and on the mobile app. Now, that's actually more complicated than you might think.
This entire product funnel, where there's multiple steps in that funnel, I'm not going to go into detail here, but if you think of a referrals product is not just as simple as throwing that offer out. This probably wouldn't work in the very beginning. But once you have a full product, you want to start measuring each of these steps. Like, what is that referral offer? How many people go to that page? How many people send invites? How many of those invites are being clicked on? How many of those people sign up from those invites? How many of those people that sign up end up booking? Each one of those steps is a step in this funnel. Let's talk about one specific step, the referrals email invite.
So, we would spend a lot of time optimizing this step, because there were lots of people getting the referrals email invite at Airbnb. So, what are the things you can optimize on the referral invite email? First, who's the sender of the email? If it's just Airbnb, I probably have never heard of Airbnb with the first time I get this email. If it's Gustaf that sent the email, and I sent it to my friends, they have heard of me. That's the reason to open the email. So people open the email. Clear value. What is this email about? Many emails just start with text? Don't start with text. Just have the clear value prop at the top. Why should I care about this email? In this case, it's extremely simple.
Gustaf sent you $40 for your first trip. That sounds good. What is that about? I'm gonna read about it. When do I have to care about this? By this date, in the next month? So I can't just leave this email and never open it again. I have to do it right now. What do I have to do here? Well, I could sign up, which is an undefined thing that you can do sometime in the future, or I can do what we did here, was accept my invitation. That sounds more exclusive. It sounds like something that is just for me. It doesn't sound like something I can do anytime in the future.
And finally, here's some social proof from this email. This is me, I live in San Francisco, we can reveal that. I actually have been a member of Airbnb since 2009, and we can reveal that as well. Let's talk about paid growth. Each of these sections, referrals, pay growth, SEO, could be a presentation on his own. So it's impossible for me to go into deep details on this. But if you're determined that you have product-market fit, you want to go on these channels, and this is the channel you want to go deep on, you're gonna have to go really deep on it, because being really good at one of these channels require a lot of work.
So, there's lots, lots stuff online about how to get really good at one of these channels. It doesn't really make sense to get good at all of them, as most of you won't really need all of them. The number one lesson in paid growth i.e. online marketing, is to not do it unless you have revenue. This is the most common mistake that founders make, is they somehow start buying ads for products, and they'll never be able to pay them back. Don't do that. The next thing you want to figure out is what's called CAC, customer acquisition cost. How much does it cost to acquire a new paying or new valuable customer? Someone's giving you value back.
Many of the advertising tools like Google and Facebook have a very clear system for how they calculate this. And once you start running ads, they'll start telling you what the cost is going to be. Next is going to be that your revenue, predicted revenue from this user, is going to have to be higher than the CAC, higher than the cost. Very simple. Otherwise, you can't do this. So, how do you know? This is the common question you get early on in paid marketing. Well, it seems like in eight months, it will be higher, but not in the first month. Well, you can't take all your money and spend on something that you have no clear certainty of is going to happen in the future.
So, you're gonna have to either wait eight months, or you're going to look for early indicators that your hypothesis about the value is going to be stronger. The best thing a startup can do is don't wait eight months. Just have a much lower target on what your CAC is going to be, maybe one month, two months, three months, first transaction, something like that. That's a much better way to do it. The main channels for online marketing these days is going to be Google, Facebook, Instagram. That's pretty much it.
Let's talk about search engine optimization. This has changed a lot in the last couple of years. It's very competitive. And what changed is there used to be millions of websites that each would rank for tens of millions of keywords. Now, what have changed is that the really big companies suddenly are getting really good at ranking for all those keywords. So, a Pinterest or a TripAdvisor might rank for every single travel keyword that you can imagine. That's hard for small companies. What that means that if you are going to rely on search engine optimization to grow, you're going to have to be as good as Pinterest or TripAdvisor. Eventually, not right away, but eventually, because it's so competitive to win in this, like, large world of SEO. When you get started, you can think of this way. SEO is a zero-sum game. Basically, you're competing against others. So what you do in SEO is going to be a matter, what you compare to others. The second thing is that the keywords that people search for are changing constantly. So, if you're building something new, let's say ASMR, I think was a thing that came up recently, lots of companies able to rank for that, because it's a new keyword. There weren't websites built 10 years ago that rank for that, because the thing didn't really exist.
All right, let's talk about SEO, how it works on the technology side. This is the Airbnb search results page. This is what you and me see when we go to the Airbnb. What does Google see? Google just see texts. So, to be good at SEO, you need to understand what text am I showing to Google, so Google can understand what the website is about. If Google can't understand what your site is about, it's not going to rank it. What are the two main leverage for SEO? The first one is going to be things I do on my page. So, for example, what's the title of the page? Can Google read that, the page? Does the page throw errors? What specific keyword am I trying to rank my page for? Well, start with the keywords, do some research, and see what are people searching for? How many people are searching for ASMR in United States per month? Maybe I want to try to rank for that keyword. Well, build a website that's trying to rank for that keyword. Start with Google. Don't start with your own content. You don't know exactly what people searching for. You're going to start doing some research.
The second thing is the thing you can't do that much about, which is called off-page optimization, or domain authority, or something like that. Which basically means, how valuable does Google perceive your website to be in the grand scheme of all websites? And the more inbound links you get from press, the more links you get from all kinds of people that are also, like, have high authority, the more valuable your website will be in the eyes of Google. Which means it will rank you higher on some of the keywords you're trying to rank for, because it will compare you to other websites and see if they seem more or less authoritative. I'm going to details here, well, that's basically how Google work. If you're curious about this, you can google PageRank and go to the Wikipedia article on PageRank. Basically, will explain sort of, like, high level how Google works.
Final section. I'm going to go through this one a little faster. Most of you guys don't have to focus on A/B testing at all. It won't matter for a long time. It is a great decision-making tool later on. Here's the situation that startups tend to get into. I want to launch a new home page, I want to launch a new design. I did, and the numbers went down. What happened? It's a really hard problem to launch something new, and then sort of, like, just look at the metric over time. Don't do that.
There's a better way. First, before you get into that stage, you want to figure out, "Is A/B testing something I want to do?" The best way to do that is to go to Google and type in "A/B testing calculator." Think of the metrics that you're trying to change here. So, like, visitors to some conversion metric, put them into the first link you see on Google, and they'll tell you whether it's going to be worth doing. Most of you, it won't be worth doing for quite a while. So, here's the example I trying to give you on the website. So, I want to ship a new experiment, or a new design on the home page. So, let's ship it. The metric went up, or the metric went down. Either way, like, I don't know if the website actually caused it or not.
The only way for me to know if this new design actually changed the metric, is if I had an alternative side of history. Like two different parallel universe at the same time, one with the new design and one with the old design. If I had that, I can tell exactly what happened. That's the definition of A/B testing. You basically have two different parallel universes of the thing you shipped at the same time, and you measured the metrics that matter to you. The reason this is so powerful, it helps you make decisions at scale.
What ends up happening to founders when they get 5 or 10 people in their company and they launch a new design, and they're arguing about what caused the thing to go up or go down, the only way to really know is to run an A/B test to figure out what does the metric say about when it went up or down? This is hard to internalize, because most people think of themselves as good product thinkers. So, I wanna talk about one thing called experiment review. We did that, this is Airbnb. How many here think that you guys have good product instincts? Raise your hand. You guys are founders. You have, you should have good product instincts.
All right, let's see as I go there. All right, so I'm going to give you two examples. So, at Airbnb, we launched a new sharing sheet for the mobile app. And this was the old version, which was the native share sheet that you've seen on iOS. You click on Share, and then you see a bunch of sharing options. And then we just tried this new share sheet that showed more options, call this the experiment, but it didn't really look native. So, the question was, which one was better? Well, we didn't really know, so we launched an A/B test. We launched both of them at same time for different users. The goal here was to measure number of shares. So, all right, how many here think that the control was better? How many people think that this experiment was better? How many people thought there was no difference? So, it's quite common. This is about 40% better for us.
So, thank God we ran experiments, because if this was the decision-making group, then we wouldn't have made the right decision. Next one, should we have a signup wall or not in the app? Now, these are not necessarily learnings that you can apply to your companies right away, but it was an important decision for us to determine. So, should we have people just open Airbnb up and go straight into the app? Or should we have an experiment where you can click out and X out the signup wall? Or should we have a sign up wall that is a wall that you can't climb over? You have to sign up, otherwise, you can't use the Airbnb app. Which one is better?
How many here think that the control, no signup wall is better? Raise your hand. How many people thought that the experiments where you can X out and then sign up was better? How many people thought that just, like, this wall, you can't climb over is better? That's a few as many people. All right, so this is a lot better. We got 2.6% more bookings from iOS by making people sign up through a sign-up wall in the app. Why is that? Well, we knew something about them, so we can show them more personalized stuff. And when they were about to book, we already had them signed up, so didn't have to, at the time of booking, go through the motions of signing up.
You want to learn why it caused this? The whole point is basically, when you get big enough, when you're starting to grow, and you have these decisions about, should I launch this thing or not, this is a really good way to do it. Product decisions are really hard. So, using data to make them is a good way. Most of you won't have to worry about this for a while, so don't worry about it. Here's the summary of my talk today. Most of you need to do things that don't scale. You are not at the place where you can think about real growth things that growth teams do.
So you have to unlearn the things you've learned at your big companies or in MBA programs, and just do things that don't scale. Secondly, you want to measure your retention to understand if you have product-market fit. There are other ways too, but that's the best way in my opinion. And third, you want to build a culture of experimentation. You want to use data, and not have the loudest voice in your room decide what the best decision is. But you want to use data and experimentation to decide, what is the best decision? Probably doesn't matter right now, but it will matter at some point. Thank you. Do we have time for questions? All right. A couple of questions. Yes? Okay.
Woman 1: So, for marketplace startups, at which stage do we have to start measuring the product-market fit? Because I think you mentioned that surveys are, can be biased, but only paid users will be accurate to measure proper product-market fit. So, at which stage in our startup should we measure?
Gustaf: The question was, if I'm a marketplace startup, at what stage should I be measuring product-market fit? So, just think about what the metrics might be. The first metric is probably on one side of the marketplace, the supply, how often are they coming back or doing the thing that they're doing? And then the other side is demand, how often are they coming back and doing what they're doing? Hopefully these are not like Airbnb was, like, once a year on the demand side. It's more frequent than that. Because then you can measure right away. There's no need to wait.
In the case of Airbnb, where supply would happen every month or every couple of weeks, like, you certainly want to measure and see if your hosts are sticking around. If they're sticking around, and they're doing their hosts books repeatedly, that means there's something good there. On the demand side, depends on how long it will take to get the second booking on demand side. But I would measure both of them, like, pretty early. Like, if I was an investor and I would meet with an early marketplace startup, I would want to understand, what is the repeat usage on either side? That's the thing I really care about. So, I would want you to measure right away. Now that measurement is not going to solve the doing things that don't scale. So, this is like a thing you do in the background while you're doing all the things that don't scale. Yes?
Woman 2: Quick question. Local search engine optimization versus national, what are the key differences? If our focus is very much on the local, what should we care about more compared to national?
Gustaf: So the question is, what matters more for local search? I wouldn't be able to tell exactly what matters. The Google algorithm is a black box, so it's hard to say. The most important thing that I would say that matters to Google is to really think about what intent and intentions does the searcher have when they put something into Google? So, if I look for apartments, apartments in San Francisco, what can I be meaning?
Well, I could be meaning renting apartments long term, I could be meaning short term rentals, I could be meaning a lot of different things around apartments. I can be meaning like the concept of apartments. The more you understand of the intention of the people that are typing in specific keywords...so if I type in "coffee shops on Mission Street," that's probably high intention that they want to go to a coffee shop on Mission Street. So, I think that's the first thing you wanna really figure out, is, like, what keywords have that high intention for the things that I offer?
Woman 2: Like a local culture or local activities...
Gustaf: Yeah, so local activity is like, if someone says, like, "What are fun things to do in Berkeley?" Or, "What are the art shows in Oakland?" Like, that was a clearly people that live in those cities that this is going to be most relevant for. And you want to rank for all of it, because you're not going to sit in Florida and type in those two things, most likely. So, I would just try to understand, what are the keywords that matters to you? And what is the intention behind those keywords? Yeah, in the back.
Man 2: On the topic of marketplaces, if my users are buying at different rates, some users are buying once a week, some users once a month, once every few months... Just to give an example, I run a website that connects graphic designers to creative entrepreneurs for art work, and some people need that once a week, once a month, like I say, for a user retention, how do I measure that for product-market fit if everyone's kind of buying at different rates?
Gustaf: So, the question was, the marketplace for people buying graphic design. How do I measure the product-market fit on each side if people have different sort of like cycles through which, when they buy that? Well, first I would try to figure out who are these different people that have these different cycles? Like, are they different kind of companies? Is there one group I rather focus on than the other? If I were Lyft, I rather focus on the people that travel daily than people that travel yearly. So that makes more sense for me to spend time on trying to acquire more of those people. So, in your case, I would try to figure out, what's the intersection between the most valuable people that are the most frequent.
It's rare that you'll have massive differences in retention and they both love your product, if that makes sense. It's very rare that you have people who use Uber once a year, and then you have lots of people use it every day, and somehow that you should be focusing on both. If you have people using it once a year, if you're Uber, your goal will be to get them to use you more often. So, I would try to think of, like, what is the ideal use case that how often they will be using what I'm doing? How often do startups need graphic design? And if they only need it once a year, you probably not gotten the product to a point where you thought of all the use cases that they would need for graphic design.
They're probably a lot of other things, everything from, like, logos to website designs to landing pages to stickers to, like, gazillion things. Like, if the more of these things that you can incorporate into your offering, probably you get that more frequent use case. But I would measure both on the supply side the designers, how many jobs are they taking on my marketplace? Are they stop to take jobs? If so, why? This is typically in the early days, a great source of ideas for a roadmap. Why did people stop using my product? Talk to them, call them, email them. Do anything you can to understand why someone stop using your product, because that's the things that you have to fix. Yes?
Man 3: In terms of the startup Airbnb, especially because it's like a new platform for a bed and breakfast, sort of like a hotel...how was the difference between experience of people, some who try Airbnb and be like, "Oh, air bed and breakfast is not that good for me" versus, "Oh, this specific host is not good for me," because the variance might be very, very high.
Gustaf: So the question was, if the quality or the variance of a product early days is very different for different people, like, how was that for Airbnb?
Man 3: For different sellers because you have different sellers in Airbnb, right? So, the host will giving quite different experience.
Gustaf: So yeah, the question was, like, how do you deal with the fact that hosts give different quality experiences, or different experiences. Well, in the marketplace, you want to make sure that you deliver consistent experience. So the best way to do that is introduce some kind of reputation system. So, it can be one of the best decisions early on, is to have the money, the payments, go through the platform. And once you've paid, you're allowed to give a review, but only then.
So only people that stayed with someone and paid can give a review. Once that you have that review, you can start weeding out bad hosts and promote the good ones. That is the best way that I can give you that will deliver consistency of experience. Now, a new host, how do you know if it's going to be consistent? You don't really know. So, a new driver on Uber, you don't really know. You have to do the best you can to onboard them. Two more questions in the very back.
Man 4: So, for example, I have a product, that when you log in, it looks pretty mature, I can charge for it. So, and I'm kind of balancing, like, charge for it right off the bat, or just put it up...
Gustaf: So the question is, I have this product. I could charge for it right away, but I'm not sure if I want to do that because that might turn away more people. I would say, the true...you should probably charge for your product right away. Because otherwise, you're going to build for people who prefer the free version. And you're going to build the features that the people that prefer the free version would want, which isn't necessarily the people that value the product.
So a good example here might be if I built something that can both work for consumers and companies, should I charge for it? Well, consumer is going to be much less likely to pay for it than companies. So, companies are the more valuable ones. So you should focus on the features that you want to build for the more valuable customers. So I would argue you should probably charge right away. And if you don't charge for it, it's kind of asking your friend, "What do you think about this product?" "Oh, it's great."
Man 4: My question was more along the lines of I wanted to charge right away. But I just put it out there as-is, or, like, meticulously testing, to make sure when somebody signs up, they stick?
Gustaf: Oh, so the question was, should I charge for it right away? You are gonna charge for it right away.
Man 4: I could launch today, or I could launch it, like, two week from now, and it's a lot more refined, so that the people that signed up are more likely to stay?
Gustaf: So the question is that, when should I launch, basically? Because I have more ideas and things I want to work on this? Most people launch too late, and not too early. And if you launch too early, like Kat said, you have another, many, many opportunities to launch again. The launch is not this big dramatic moment where you get judged by the entire world, and if they say no, then they'll never come back again.
Man 4: I think, I've never seen a company dying from losing customers in the first two weeks.
Gustaf: Never seen a company die from losing customers in the first two weeks. Basically, you should launch right away. That's the answer. And if it doesn't work, launch again. Last question, right there.
Man 5: Hardware company, and we have a crowdfunded campaign, so we're essentially taking pre-orders. We can't optimize for retention yet, so there's no product-market fit and the product isn't shipping till December. We have about 500 orders right now. What do you think about adding, like, paid acquisition? Because I am getting, like, revenue, basically...
Gustaf: So, the question is, I have a hardware product. I haven't launched yet. I'm doing Kickstarter, so I can't measure product-market fit. You don't have to worry about product-market fit at all. Because it doesn't matter. You don't really know if people are gonna use your product yet, because you haven't built it. So that's the truth here. Your goal is to launch it, but because it's more expensive than just a website, you're gonna need a lot more money to launch it. As long as the CAC work out, that you are paying some money to acquire people through kickstart, and you get more money back from those people, you could do paid.
You're going to have to be very good at understanding those two things, measuring the difference between the CAC and LTV, and I don't know to what extent Kickstarter provide the tools for you to measure specifically the people that came in through your ads and the ability to measure the return on investment for those people, but that's the thing you really need to figure out. I think it could be fine if you know what you're doing. I think it's risky to spend money on boosting a Kickstarter product. I think what I've seen on most of the successful ones, what they do is to try to get their friends early on in the launch, so they have momentum right away, and then momentum itself will carry on the Kickstarter campaign. That's what I've seen. All right, thank you very much.