I’d like this blog to be a mix of thoughts/opinions and actionable useful startup advice.
Since I already have said your users (not you!) are the experts in their problem domains, and you’re supposed to be all about trying to help them, today I’d like to share a bit about surveys.
The big lesson, if you only learn one thing: Use surveys to find out what you don’t know, not to confirm what you want to do.
I think much of my dissatisfaction from major platforms and software products comes when they have never asked me what I wanted, and this pattern is all too common. Very few people ever ask, and this makes me feel they treat the customer as a source of money versus someone they want to help. They could probably start delighting users and having many more product champions if they changed their approach.
First off, when doing surveys, don’t ask “Net Promoter Score” questions that ask you to rank people from a scale of 1 to 10. These graphs are used by people to self-promote themselves at corporate meetings and don’t really tell you all that much that you can react to, and many of us can see them coming and intentionally don’t respond. I know there’s some pseudoscience around these, about when you should follow up or add people to different lists, but I just don’t like them. I have basically never gotten a manual follow up, regardless of if I’ve hit 1, 6, 7, 9, or 10. And everybody matters, regardless of how you vote.
I mentioned earlier I knew a PM that wanted to do the bare minimum to retain existing customers, and that PM and his company failed because he didn’t create any champions of the product (and a number of other reasons). The product was just good enough to have sales get some people to buy it - with a lot of effort.
You may think people don’t want to take surveys, and this is mostly true from surveys appearing to be automated junk mail. Send them with a really honest message that isn’t a bunch of business-speak.
People will help you out with questions when it’s clear you are going to use those questions to react, and it often also helps if you volunteer to share the survey results with everyone (at least, in part). This is especially true in cases where users are really enthusiastic about a product and the community feels authentic and not just a top-down relationship.
You should ask people good questions that will help you help them. Folks may think they are going to be enough to craft multiple-choice responses, but if you’re not surveying 20,000 people, open-ended questions are the best, even if it is just a few. Your goal is to find out the things you don’t know to ask, to understand nuance and how people feel.
Some good questions are questions like “What problems have you or your colleagues had learning ___?” or “What unsolved problems do you have while doing ____ and what things remain un-automated or poorly automated in your infrastructure?” or “If you could have anything at all, what features would you most be looking for in the next release?”. Another good one is “What other tools do you use in your work place to ___?”
You’re trying to identify gaps in your product/service and ways for attach to more users during your onboarding experience.
Multiple-choice questions can be good for gathering information about what platforms people use, but I find people mostly do want to talk to you, and don’t want to be treated like another number in the system.
Sometimes it’s important to quantify interest, because people may say they want things and be unwilling to pay for them. If considering a new product, you might want to ask a question of “how much would you be willing to pay for a solution that ___”. This isn’t so much to help you set pricing, but to decide if it’s something people would actually purchase.
Another way of looking at that same problem: people may say “yes” to everything in a list of “which of these features would you like”. I learned a pretty good trick from one of the students one college class I helped teach - if you have a list of 10 possible features, give the survey participant $25 imaginary dollars and ask them to bid for features by assigning each a dollar value. This lets you see not only which features might be the most important and by how much.
You may think that people don’t want to share their email or company, so I’ve asked questions like “If you don’t mind sharing, can you share details about where you work and what you do day to day? This could include information about the size of your infrastructure and so on”. If you want, you could even add “this data won’t be used for sales emails” if you thought it helped. I’ve learned some pretty good and valuable stories that way.
Finally, when you look at survey results, don’t just look at survey results in aggregate, take time to read each set of grouped questions one by one, and try to develop an understanding beyond the dashboards - who are your people, and what patterns do you start to see when you read through them?
Use all this to drive your roadmaps, how you talk to customers, and how you see your company. Work on making these people happy. You can’t find that understanding in a “1 to 10” Net Promoter Score question, and that can be your competitive advantage.