I think there's clear danger of having a local maximum problem here. This is because the role of hypothesis in the process seems either misunderstood or misplaced.
The Tim Robinson example in the article recounting the removing of links on the checkout flow illustrates this. He observed something interesting in the data (people clicking on links), but he then skipped straight to a test which removed links and found conversion went up. It's implied that people were being distracted from the main task by the links.
But it might equally be that they wanted to click the links to find some missing information about the product. Or even some other reason. Without any qualitative analysis to establish a reason why people were clicking on them, you don't know if he might have raised conversion even higher had he kept the links and addressed a need people had. This is particularly important in a checkout flow where the numbers of people are comparatively low and behavioural elasticity is also comparatively low.
So I'm reading between the lines here that because qualitative analysis is hard, people don't do it. Maybe that's fair enough, but that doesn't prevent you from kicking the tyres on hypotheses. Quant analysis and A/B testing will only tell you what happened, not why did.
I always love the debate on whether great products are imagined or are discovered through experimentation (certainly a bit of both). This post details the advice I usually give to startups on this topic and I'm happy to answer questions and join the discussion.
Yes, one of our YC partner Dalton Caldwell hammered this idea in my mind. He told me that when he sees a founder trying to attack a product that other have either failed or succeeded at - he looks for what they've learned from those folks. If the founder dismisses previous attempts to solve this problem or similar ones - that is a very bad sign.
Just curious growth wise, did Socialcam have a lot of word of mouth, or was some growth hacking involved as well. 16 million downloads in 3 months is incredible.
Sometimes starting the other way around might be helpful - not seeking inspiration but defining the essence of your value proposition and starting from there.
This was a better article than I expected. Finding ways to measure your assumptions is great, because if you can measure the behavior, you can measure the impact of changes.
Generally, I'm not a fan of "scientific method" as a phrase. It's too often used by people dismissing intuition in a very non-scientific way.
Thanks - I agree that balance is really important. Also, in many cases - especially where the founder is a user - intuition can be a key early driver of success.
Absolutely agree. I'm a "scratching my own itch" founder, and building a product based on years of doing the job of my target users. "Scientific method" isn't terribly useful for me as I build the initial product.
On the other hand, I don't necessarily know what this technology is going to mean to end users. They're going to find problems to solve that I didn't even imagine. As we're closing in on a beta, I'm doing demo/interviews as often as I can (a couple of times a week at this point), and seeing where their minds run. As William Gibson said, the street finds its own uses for things.
Once we get the product in the hands of real customers, it'll be far easier to start taking measurements and figuring out what to work on from there. But for now, intuition is the only tool I have.
This is one of the reasons why YC pushes people to launch MVPs so quickly - the speed of learning is much faster when the product is in customer's hands.
It's not as scientific, but you can prepare a/b tests for paper prototypes and ask your users to go through them as they would go through the site.
Quantitative methods aren't reliable with such a small dataset so you will have to rely on assumptions and guesses. Unfortunately this makes your observations prone to bias from your hypothesis, so it's especially important to be mindful of this and keep them separate.
Edit: Didn't realize this was an AMA but hopefully my answer contributes to the discussion.
Sure, it does! I just added it to register that is weird to publish as an AMA and not answer many questions.
I think your approach is right. One thing that helps me is to use hypothesis from users/visitors/customer as much as possible. Talk to customers and try to distill what are their hypothesis about a specific problem. Talking to customer goes a long way.
On the topic, I think that famous quote atributed to Ford "If I asked my customer what they wanted, I would be trying to create faster horses". I think is BS (and probably bogus). That ad absurdum scenario would only happens if he asked the wrong questions and/or followed the most superficial answers.
Its funny - I think the Ford quote is right. I always tell folks to not ask a customer what they want but ask them what their problem is. Its the customer task to describe and demonstrate their problem - its the founder's task to think up and test solutions for that problem.
About the quote, it kinds of depends on what do you mean by "want", but I understand and agree of your approach (while still disagreeing with the quote if that is possible).
BTW the quote is probably bogus, as the first Ford cars were slower than horses. 45km/h vs 88km/h (world record)
sorry for the delay :) - its really hard to do active a/b tests in the early days - often I tell founders that feedback cycle then has to rely of direct interaction with customers alone. Which I think is a good habit for all startups to keep even when they can get quantitive data.
Hi Michael!
What problem do you think a game company is solving? What would be reasonable key advantages at a company level? I.e.: What do you know/have that Zynga (or even smaller studios) don't? lol
Yeah! I agree! I asked that because I think it's hard to distinguish yourself by solving this general consumer entertainment problem. Seems that all game companies do that, even when thinking about small studios, it is hard to find differentiating factors that does not rely on the game itself.
I also agree that A/B testing is super important! We are following exactly what you have described here http://www.michaelseibel.com/blog/product-development-cycle-... but for a game, and the results are pretty interesting! Specially because we don't need a GDD (a document describing the game), all members of the team are involved in the creation process (and they feel more motivated to work - and believe me, this is big problem in indie studios) and is easier to propose changes and improvements to the game, since them just need to bring us closer to our goal with the game (for example: 20k ad visualizations / day). I'm anxious to get into this A/B testing phase! We haven't launched yet.
Thank you so much for sharing the knowledge! These posts were really valuable and helpful for me! And I think they are complementary!
I wish the article had "screenshots" to show what was being changed, and why. It would help people to visualize and apply the steps better to their startups. Excellent read!
Thanks for doing this Michael! Any advice on getting people to pay for something before it's built? I think a lot of people will say they will pay a certain price for something but when it comes to signing up once they can enter their credit card, that doesn't end up being the case. Are LOI's the main way, and does that work for smaller projects < $100/mo?
My Q: When one a/b tests there is generally a winner, but a better option might be possible if the team spends a bit more time brainstorming and hypothesizing. Do you have any thoughts on when to go with the results of the a/b test vs when it's worth going back to the drawing board to try to brainstorm a better idea?
In operating my startup or working at YC I haven't faced any overt racism (in the real world just living life that is a different story). Have I faced any covert racism? - very hard for me to tell because so much of startups is facing failure and there are often many probable causes of that failure. Not sure what you mean by the second question.
A shorter and cheaper version of this would be to create an emotional connection between your users and your product. You need to become a user and figure out if you feel accomplished after using your own product.
What are some best practices for evaluating ideas before building anything. You mentioned talking to users which is good advice but assumes that an industry has been identified already.
The first thing I ask a founder is how often does the target customer have the problem they are looking to solve and how much money are they willing to spend to solve it. The more often the customer has your problem and the more money they are willing to spend - the easier it will be (all other things being held equal) to get customers and iterate. Sequoia calls these problems hair on fire problems. The second thing I ask is who else is trying to solve the same problem and what do you know that they don't.
This is great advice and advice I wish I taken onboard earlier. As a technical founder Ive found myself making a large set of assumptions which are usually proved to be incorrect when going to the market.
Going into the market and talking to your customers early on so you can answer those two questions will really help tell you if you have a viable idea.
>>> assumes that an industry has been identified already.
This is one of the key take aways from customer discovery - is your customer actively looking for a solution to the pain / problem you've identified. If your customer is actively looking for a solution than it is easy(ier) to identify and talk with them. If no one is looking for a solution then it may be that you haven't identified a real pain, you haven't identified the correct customer, or you are to early.
It's easy to say that you can't talk to customers because the industry hasn't been identified, but that is a bad statement and should be avoided.
Ideally you'd be a user of your own product, someone who is actively experiencing the pain and is looking for a solution. Then you'd look for people like yourself.
Otherwise you still want to follow the general pattern of talk to whomever stands to gain the most from your service. Note that the person that stands to gain the most might not be who you first assume it is.
Hi Michael, how does one conduct customer research? What questions do you ask? How do you know you're doing it right? Are there any good books on customer research?
Different answer depending on whether you have a product built or not. Also different if you have a consumer product. My best personal experience with customer research is for live consumer products. In that case we would recruit potential customers off of craigslist (offering $50 for an hour of their time). We would bring them into the office and sit them at the table. The entire team (small) would be in the room. We would give them a scenario and ask them to complete a task in our app in front of us. For example, you saw a really good video on Facebook made with this app and now you want to download that same app and make a video for yourself. Then we would ask them to complete that task asking us any questions they had and sharing their inner monolog out loud. Some questions we would answer but most we would ask them to figure it out. It was very important that everyone involved with building and designing the product was in the room experiencing the user test live. We would take notes on issues the customer faced and conduct about 5 interviews in an afternoon. Anything that came up 4-5 times we would attempt to fix. We would rince and repeat monthly.
Start with better understanding what exactly customer discovery is, Steve Blank and Eric Reis are good places to start.
To better understand how to conduct the discovery and what questions to ask you should read a bit of Rob Fitzpatrick's The Mom Test.
It's hard to give one piece of advice about how you should do customer discovery (without having a high quality conversation about what you are doing) b/c every venture is a different in what needs to be tested. Software ventures tend to need product customer discovery (what you should be building), while STEM companies[1] tend to need go-to-market strategy customer discovery.
[1] An example would be a company that is making an orally delivered anti-coagulant drug. They have HIGH technology constraints that aren't easily adjusted, so their customer discovery tends to revolve around questions about what their real value proposition is (is it pain free needle free drug delivery or no medical training drug delivery or something else) and other such questions.
I always loved Mixpanel - but its been a couple years since I've been active on the product side and there are now tons of great tools. In the end of the day - use the events based analytics product that your team will enjoy looking at everyday.
We are new to the startup scenerio.We have a great customer following.But we r looking for our seed fund /guidance to start it up;Its a platform to connect ,facilitate and empower moms.What would be ur suggestion for us to march ahead ,its a real world problem we r trying to solve .We dnt have an app and an mvp to show as our business model is slightly different.what would be ur suggestion to march ahead?We want to launch asap.
The Tim Robinson example in the article recounting the removing of links on the checkout flow illustrates this. He observed something interesting in the data (people clicking on links), but he then skipped straight to a test which removed links and found conversion went up. It's implied that people were being distracted from the main task by the links.
But it might equally be that they wanted to click the links to find some missing information about the product. Or even some other reason. Without any qualitative analysis to establish a reason why people were clicking on them, you don't know if he might have raised conversion even higher had he kept the links and addressed a need people had. This is particularly important in a checkout flow where the numbers of people are comparatively low and behavioural elasticity is also comparatively low.
So I'm reading between the lines here that because qualitative analysis is hard, people don't do it. Maybe that's fair enough, but that doesn't prevent you from kicking the tyres on hypotheses. Quant analysis and A/B testing will only tell you what happened, not why did.