A recurring theme of my conversations with folks in the past week has to do with Beta products and what the goal of beta testing is really about, particularly when you are talking about bringing a new product to market. My point of view on this is that beta testing is as much a marketing exercise as it is a development exercise.
The traditional view of beta products is that Beta is about testing specific features of the product to make sure they work in the customer’s environment. A better way to think about beta testing is to think about is as a period where you are testing the set of assumptions you made about the customer problem and whether or not your product solves that problem in a way the customer understands and values. The shift in thinking has a big impact on both what you include in the beta product and what you do with your customers during the beta.
Here’s a comparison of the two approaches:
- Who Manages the Beta: Development
- Product: As feature-rich as time and money allows, but testing is incomplete. Essentially the product you plan to release at the end of beta when testing is complete.
- Data Collected: What other systems are in the environment for compatibility or integration challenges, number of failures and the conditions under which the failure occurred, scalability, and usually some general feedback around ease of the use and the UI.
- When to Exit: When internal testing is completed (i.e. the Beta runs for a set period of time unless a serious unsolved defect is found), when the full release set of features are ready (again, independent of the Beta program).
Testing Assumptions Beta
- Who manages the Beta: The Business Owner.
- Product: The minimum set of features you need in order to attract customers.
- Data Collected: How much are particular features accessed and used and to what purpose, what are the most-loved features and the biggest annoyances, would the customer pay for this product, what are the alternatives to this product, would the customer recommend the product and to what sort of person/enterprise, what to customers wish was there but isn’t, would the customer be very disappointed if you took the product away from them. For online products you are looking at things like signups and conversions in addition to the above (this isn’t a complete list but hopefully you get the idea).
- When to Exit: When the business owner feels confident that you’ve got a product that there is a real need for, that customers love and are willing to pay money for.
Notice how the first scenario looks like a development exercise and the second one looks more like a product marketing exercise.