BUYER BEWARE: No segment filtering in test results? Not an effective testing or targeting tool.
A critical step in the testing and optimization process is filtering your reports by customer characteristics, uncovering how your tests perform in smaller groupings or segments of your visitors. A testing tool that does not provide this feature does not allow you to interpret your test results or target fully and accurately. You will not clearly see how your content is performing in different sub-sections of your customers. You are also completely incapable of qualifying your test results based on other influencing factors within your test group, making it impossible to target relevant content. It is surprising to hear that there are testing tools out there, right now, that do not provide this option; be sure to inquire and evaluate these capabilities before choosing to implement a new product. Let me share the impact of not having this capability.
Here’s a simple A/B test example that clearly underlines the issues: I’m an online clothing retailer. Let’s say I’m testing my home page masthead banner ad. One option has a “buy one, get one free” offer on shirts, and the other offers “free shipping on orders over $50”. Let’s take it one step further and say that I only want to test these offers within the customer segment of California residents. I target the test to California residents. After reaching significance, I find that 85% of Californians prefer the “buy one get one free” offer. It might seem logical to then target the “buy one, get one free” offer to Californians, and call it a day.
BUT………..what if we filter these test results by the segment of gender. Or age. Or category affinity. We just might find that California women in their 30s actually prefer the “free shipping on orders over $50”. If we had targeted the “Buy one, get one free” ad to all Californians based upon the unfiltered results, California women would not be receiving their preferred, most relevant ad!!! Even if I ran an additional test targeted to California women, I could not fully evaluate my original test results based on these new findings.
I can’t stress enough how important it is to evaluate your results completely, based on the critical segments or subdivisions of your digital population. There is no other way to interpret your results fully, from every angle and every influencing factor, without this capability. When shopping for an effective testing and targeting tool that will be an integral, long-term investment for the growth of your optimization program, a tool that does not provide segment filtering in their results is not worth your time or evaluation. You need a tool that will adapt and grow with the changing needs and influences of your customers over the long-term, not a tool that gives you misleading results.
Adobe Test&Target has segment filtering capabilities in its reports, with built-in segments right out of the box. Segment filters can also be customized based upon any influencing factor that defines a segment in your population. New segments and segment filters can also be generated based upon any customer data or attribute captured in mboxes, as well as from 3rd party data integration. In addition, being an open platform, with SiteCatalyst integrations and built in 3rd party data aggregation, user profiles can undergo an enrichment process, bringing a sophistication to the segmentation and targeting process. With all of this at your fingertips, no influencing factor will be overlooked, and you can feel secure that there won’t be substantial portions of your visitors who are left receiving the wrong content at the wrong time.