Last time around, I promised an in depth look at how we measure campaigns in Test&Target as a way to illustrate state-based measurement. Today I deliver. My goal is for you to understand how the application works, why campaigns are an important measurement, and why we measure them the way we do.
However, before we dive into the measurement, you’ll need a basic (and I do mean basic) understanding of how Test&Target works. If you’re already a Test&Target customer or you’re familiar with what it does and how it works, skip down to the What’s Important section.
For the official version of what Test&Target actually is/does, check out the Omniture site, but here’s my over-simplistic version.
Test&Target is an A/B testing, multivariate testing, and content targeting tool that allows you to try out different versions of content/creative/offers on your site. You can also see which segments of your audience respond to the different versions, then target the top performing versions to the segments that responded. You can manage each step along the way, or put in your different versions of content and put it on auto-pilot.
Each campaign that you run can have one or more experiences you want your audience to see. These experiences can be on one page or span multiple pages on your site. They can be targeted to a fixed percentage of your audience, a specific segment of your audience, the entire audience, or some combination of the above.
When your pages load, a request is made to Omniture. Omniture determines which version of content (if any) to show to that visitor based on the way your campaign is set up. Then the content (or offer in Test&Target lingo) is returned to the page.
Omniture is not a retail shop (shocking I know), our revenue is driven by subscription contracts. For Test&Target, those contracts are for Mbox Requests. Mboxes (marketing boxes) are the areas of the site where Test&Target will place the alternate content. Obviously we want to measure the number of Mbox Requests, but those are transactional and while the data itself is interesting, there’s nothing inherently fascinating about the measurement method.
Active Campaigns, however, is state-based and much more interesting in terms of how we measure it. An Active Campaign is a campaign that is live and running right now on a Customer’s website.
More Active Campaigns means tests are running, offers are being served, visitors are getting relevant content, and customers are seeing value from the product (all good things). Fewer Active Campaigns means the opposite. If a customer is not running many campaigns, that indicates they’re not using the product. If they’re not using the product, it’s a safe bet that they’re not getting the promised value, which is something we like to avoid.
If a customer was running 22 campaigns yesterday and launched 2 more this morning, then our measurement will reflect the additional 2 campaigns and a growth rate of 9.1%. Similarly, if 3 campaigns were turned off or completed, we would see the attrition rate (or negative growth rate if you prefer) of 13.6%.
State-based metrics are inherently different than transactional metrics because we can’t just add them up to look at them. We’re looking at these as of a point in time, and then trending them over time to see growth and attrition. So we take a measurement every day and record it. We probably won’t look at it every day, but daily granularity lets us aggregate to weeks, months, quarters, and years as needed. If we only had weekly data we couldn’t aggregate properly. Daily granularity also preserves our ability to drill into the data when we see abnormalities at the weekly or monthly level.
So where do we actually measure this? For this kind of thing, we rely on plain old databases and timed processes (cron jobs for the technical folks in the audience). At the end of every day, when all the day’s data has rolled in, there is a computer process to run a few queries on the back-end databases to collect the data we want. There are a number of data points we collect and one of them is Active Campaigns per Customer.
Once the data is in our hands, there are a few options at our disposal get the data into SiteCatalyst. You can use the standard XML-based Data Insertion API, the new PHP measurement class, or a Data Source of some kind. Implementation will obviously vary depending on which route you go. I went with the PHP measurement class in a timestamped report suite, picked a conversion variable (eVar) to hold Customer Name, and picked a conversion event to hold Active Campaigns. We must also send in a Day Counter event (1 event per day) and I’ll get to that in more detail in a moment.
We then turned the Active Campaign event into a Numeric event instead of a Counter event (through the Admin Console). Numeric events can count up by arbitrary numbers that you pass in, rather than just counting by one. That means I spend only one server call per customer even if they’re running 25 campaigns. These processes are all run automatically in the wee hours of the morning and all the data is processed and ready to be looked at by the time I roll out of bed.
There are a few things to keep in mind when you report on this kind of data, since it does differ from standard transactional data. First and foremost, you must always keep granularity in mind.
For an example, above, is the number of Active Campaigns that one of the Omniture marketing teams is running for the month of October (to see the results of some Omniture campaigns, you can play this game). Keep in mind the state-nature of the report, and you can see that they’ve got around 100 campaigns running each day with some growth in the latter end of the month. If I look at this on a weekly basis then it would tell me that there are around 700, which is obviously not correct (in addition to the misleading partial week).
Monthly suffers the same problem, but to the tune of 30k Active Campaigns. In fact, any time period greater than a day dramatically inflates the numbers. If this is all we had, our analysis potential would be rather limited, but not to worry. To the rescue comes the Day Counter event that I mentioned a moment ago. By creating a calculated metric that divides Active Campaigns by Total Day Counters (700 / 7 or 31,000 / 31), we get an Avg Active Campaigns metric that can be freely applied to any time frame.
Looking at the same report with my Avg metric yields the following:
Much better. Much easier to see the overall trend here with the growth displayed nicely (and accurately). The number of Active Campaigns this team is running is increasing over time. The fact that they are running more and more campaign indicates to me that they are realizing the value of the product and are trying different tests to increase their ROI. My hope is that as they complete one test they get some interesting results which drives one or more additional tests to increase learnings about their audience or to target particular content to a segment of their audience
If you have any questions about how state-based measurement could apply to your business or need some help interpreting the results, feel free to email: ben / dot / robison / at / adobe / dot / com. You can also find me on Twitter (http://twitter.com/ben_rob).