A recurring theme of my conversations with folks in the past week has to do with Beta products and what the goal of beta testing is really about, particularly when you are talking about bringing a new product to market. My point of view on this is that beta testing is as much a marketing exercise as it is a development exercise.
The traditional view of beta products is that Beta is about testing specific features of the product to make sure they work in the customer’s environment. A better way to think about beta testing is to think about is as a period where you are testing the set of assumptions you made about the customer problem and whether or not your product solves that problem in a way the customer understands and values. The shift in thinking has a big impact on both what you include in the beta product and what you do with your customers during the beta.
Here’s a comparison of the two approaches:
Traditional Beta
- Who Manages the Beta: Development
- Product: As feature-rich as time and money allows, but testing is incomplete. Essentially the product you plan to release at the end of beta when testing is complete.
- Data Collected: What other systems are in the environment for compatibility or integration challenges, number of failures and the conditions under which the failure occurred, scalability, and usually some general feedback around ease of the use and the UI.
- When to Exit: When internal testing is completed (i.e. the Beta runs for a set period of time unless a serious unsolved defect is found), when the full release set of features are ready (again, independent of the Beta program).
Testing Assumptions Beta
- Who manages the Beta: The Business Owner.
- Product: The minimum set of features you need in order to attract customers.
- Data Collected: How much are particular features accessed and used and to what purpose, what are the most-loved features and the biggest annoyances, would the customer pay for this product, what are the alternatives to this product, would the customer recommend the product and to what sort of person/enterprise, what to customers wish was there but isn’t, would the customer be very disappointed if you took the product away from them. For online products you are looking at things like signups and conversions in addition to the above (this isn’t a complete list but hopefully you get the idea).
- When to Exit: When the business owner feels confident that you’ve got a product that there is a real need for, that customers love and are willing to pay money for.
Notice how the first scenario looks like a development exercise and the second one looks more like a product marketing exercise.
Subscribe to this blog or follow me on Twitter or Friendfeed
Love the concept, but are you arguing that the marketing exercise occurs instead of or in parallel with the traditional beta? In either case, shouldn’t the marketing exercise be accomplished earlier on in the process with low-res wireframes and conceptual process flows instead of developed code? I’d rather have that product marketing data before a developer committed any of the requirements into a product. I am hesitant to abandon the traditional beta concept, and I am sure that QA/Development would be as well.
Hi Len,
Thanks for the comment. I’m saying it happens in parallel. You’re right for certain types of product you might be able to get away with wireframes and process flows to get you the info you need to test your assumptions. For others, you are going to need to have something that looks more like an early product. Either of those is better than nothing at all, which is what I’m still seeing a lot of – meaning the product marketing “data” is full of untested assumptions when the product goes into beta and that beta program is only testing code quality.
April
Blog Post-Beta as a Product Marketing Exercise : http://tinyurl.com/ya5czb5 #prodmgmt
Beta as a Product Marketing Exercise : http://tinyurl.com/ya5czb5 -> interesting perspective in this blog post #prodmgmt
RT @aprildunford: Blog Post-Beta as a Product Marketing Exercise : http://tinyurl.com/ya5czb5 #prodmgmt #prodmktg
RT @aprildunford: Beta as a Product Marketing Exercise http://bit.ly/8BgVva
Great post. I recently moved from a hardware world (high end analytical measurement and test equipment) to the enterprise software world, and the disparity in the expected beta testing is pretty dramatic.
The second scenario you describe is pretty much my whole realm of expertise. I, as the business owner (product manager) was wholly responsible for driving the beta, from finding the targets, to negotiating the conditions (usually you don’t “give away” $250K or so of hardware) and manage the expectations on both sides of the equation.
Unfortunately, corner office occupants often hamstring any true value of a real beta when they commit a delivery date that destroys the schedule.
The problem I have always experienced is that getting the product in a shape to be shared with a beta partner, usually is close enough to the final release, that apart from some early feedback and minor performance/usability tweaks, there really isn’t the time to go through a cycle of revision based on any feedback.
This has forced me (and my near brethren) do do a lot more up front research, and validation prior to formal program launching, and revalidation during the development process. Usually HW development cycles are long enough that we can bring in key opinion leaders several times to revisit.
I am getting close to my first beta in the SW realm, so hopefully I will see a different pattern of experience.
Geoff
Hi Geoffrey,
Thanks so much for the comment. You make a great point about the differences in how this works with different kinds of products. A lot of the examples people are talking about online are related to pure web-based businesses which gets you further away from what you describe in a hardware-based business. It’s different as well when you are talking about first release products versus updated and ongoing releases. In a perfect world though, when we are talking about a new product we should be willing to do whatever it takes to make sure we’ve got good fit between the product and the market before we are done with it.
Good luck with the beta!
April
April,
Just wanted to say that the new theme and the new Blog engine are awesome.
Hey, thanks so much! It was long overdue – I wish I had taken the time to move it a year ago. 🙂
April
Commented on blog post from @aprildunford – Beta as a Product Marketing Exercise : http://tinyurl.com/ya5czb5 #prodmgmt #prodmktg
Excellent post, and some great comments. With the rise of social media, I also think beta testing creates some great promotional opportunities. If anyone’s interested, I expand on my thoughts here: http://bit.ly/7IXX2D (In addition to being a product marketing exercise, beta testing can also be a great marketing opportunity.)
Thanks for the comment. I agree with your post in that having folks talking about their experiences with your product can be great marketing.
April
@aprildunford Agree w/what to test for prod mktg but don't wait until beta! Too late if you were wrong! http://tinyurl.com/ylznwm9 #prodmgmt
Beta as a Product Marketing Exercise – http://bit.ly/5Krm5K #leanstartup
RT @dcancel: Beta as a Product Marketing Exercise – http://bit.ly/5Krm5K #leanstartup
RT @teresalawlor Beta as a Product Marketing Exercise : http://tinyurl.com/ya5czb5
Test out @readytalk's Beta phas… http://bit.ly/a8tARk
[…] I read a nice post by April Dunford on her Rocket Watcher blog about her take on beta programs. I also found a rant by Tom Grant on beta programs with an […]
Beta as a Product Marketing Exercise | Rocket Watcher Product Marketing for Startups: http://is.gd/i2SwD
RT @rodpaolucci: Beta as a Product Marketing Exercise | Rocket Watcher Product Marketing for Startups: http://is.gd/i2SwD
RT @rodpaolucci: Beta as a Product Marketing Exercise | Rocket Watcher Product Marketing for Startups: http://is.gd/i2SwD
RT @aprildunford: Beta as a Product Marketing Exercise http://bit.ly/8BgVva
Can you use your product's Beta as a Product Marketing Exercise? http://t.co/huf7azmP
Can you use your product's Beta as a Product Marketing Exercise? http://t.co/huf7azmP
Can you use your product's Beta as a Product Marketing Exercise? http://t.co/huf7azmP
Can you use your product's Beta as a Product Marketing Exercise? http://t.co/huf7azmP
RT @techsparks2012 Can you use your product's Beta as a #Product #Marketing Exercise? http://t.co/T6P9gnfZ
RT @techsparks2012 Can you use your product's Beta as a #Product #Marketing Exercise? http://t.co/T6P9gnfZ