Last time around, I promised an in depth look at how we mea­sure cam­paigns in Test&Target as a way to illus­trate state-based mea­sure­ment.  Today I deliver.  My goal is for you to under­stand how the appli­ca­tion works, why cam­paigns are an impor­tant mea­sure­ment, and why we mea­sure them the way we do.  

How­ever, before we dive into the mea­sure­ment, you’ll need a basic (and I do mean basic) under­stand­ing of how Test&Target works.  If you’re already a Test&Target cus­tomer or you’re famil­iar with what it does and how it works, skip down to the What’s Impor­tant section.

Test&Target

For the offi­cial ver­sion of what Test&Target actu­ally is/does, check out the Omni­ture site, but here’s my over-simplistic version.

Test&Target is an A/B test­ing, mul­ti­vari­ate test­ing, and con­tent tar­get­ing tool that allows you to try out dif­fer­ent ver­sions of content/creative/offers on your site.  You can also see which seg­ments of your audi­ence respond to the dif­fer­ent ver­sions, then tar­get the top per­form­ing ver­sions to the seg­ments that responded.  You can man­age each step along the way, or put in your dif­fer­ent ver­sions of con­tent and put it on auto-pilot.

Each cam­paign that you run can have one or more expe­ri­ences you want your audi­ence to see.  These expe­ri­ences can be on one page or span mul­ti­ple pages on your site.  They can be tar­geted to a fixed per­cent­age of your audi­ence, a spe­cific seg­ment of your audi­ence, the entire audi­ence, or some com­bi­na­tion of the above.

When your pages load, a request is made to Omni­ture.  Omni­ture deter­mines which ver­sion of con­tent (if any) to show to that vis­i­tor based on the way your cam­paign is set up.  Then the con­tent (or offer in Test&Target lingo) is returned to the page.

What’s Impor­tant

Omni­ture is not a retail shop (shock­ing I know), our rev­enue is dri­ven by sub­scrip­tion con­tracts.  For Test&Target, those con­tracts are for Mbox Requests.  Mboxes (mar­ket­ing boxes) are the areas of the site where Test&Target will place the alter­nate con­tent.  Obvi­ously we want to mea­sure the num­ber of Mbox Requests, but those are trans­ac­tional and while the data itself is inter­est­ing, there’s noth­ing inher­ently fas­ci­nat­ing about the mea­sure­ment method.

Active Cam­paigns, how­ever, is state-based and much more inter­est­ing in terms of how we mea­sure it.  An Active Cam­paign is a cam­paign that is live and run­ning right now on a Customer’s website. 

More Active Cam­paigns means tests are run­ning, offers are being served, vis­i­tors are get­ting rel­e­vant con­tent, and cus­tomers are see­ing value from the prod­uct (all good things).  Fewer Active Cam­paigns means the oppo­site.  If a cus­tomer is not run­ning many cam­paigns, that indi­cates they’re not using the prod­uct.  If they’re not using the prod­uct, it’s a safe bet that they’re not get­ting the promised value, which is some­thing we like to avoid.

If a cus­tomer was run­ning 22 cam­paigns yes­ter­day and launched 2 more this morn­ing, then our mea­sure­ment will reflect the addi­tional 2 cam­paigns and a growth rate of 9.1%.  Sim­i­larly, if 3 cam­paigns were turned off or com­pleted, we would see the attri­tion rate (or neg­a­tive growth rate if you pre­fer) of 13.6%.

Mea­sure­ment Method­ol­ogy

State-based met­rics are inher­ently dif­fer­ent than trans­ac­tional met­rics because we can’t just add them up to look at them.  We’re look­ing at these as of a point in time, and then trend­ing them over time to see growth and attri­tion.  So we take a mea­sure­ment every day and record it.  We prob­a­bly won’t look at it every day, but daily gran­u­lar­ity lets us aggre­gate to weeks, months, quar­ters, and years as needed.  If we only had weekly data we couldn’t aggre­gate prop­erly.  Daily gran­u­lar­ity also pre­serves our abil­ity to drill into the data when we see abnor­mal­i­ties at the weekly or monthly level.

So where do we actu­ally mea­sure this?  For this kind of thing, we rely on plain old data­bases and timed processes (cron jobs for the tech­ni­cal folks in the audi­ence).  At the end of every day, when all the day’s data has rolled in, there is a com­puter process to run a few queries on the back-end data­bases to col­lect the data we want.  There are a num­ber of data points we col­lect and one of them is Active Cam­paigns per Customer.

Once the data is in our hands, there are a few options at our dis­posal get the data into Site­Cat­a­lyst.  You can use the stan­dard XML-based Data Inser­tion API, the new PHP mea­sure­ment class, or a Data Source of some kind.  Imple­men­ta­tion will obvi­ously vary depend­ing on which route you go.  I went with the PHP mea­sure­ment class in a time­stamped report suite, picked a con­ver­sion vari­able (eVar) to hold Cus­tomer Name, and picked a con­ver­sion event to hold Active Cam­paigns.  We must also send in a Day Counter event (1 event per day) and I’ll get to that in more detail in a moment.

We then turned the Active Cam­paign event into a Numeric event instead of a Counter event (through the Admin Con­sole).  Numeric events can count up by arbi­trary num­bers that you pass in, rather than just count­ing by one.  That means I spend only one server call per cus­tomer even if they’re run­ning 25 cam­paigns.  These processes are all run auto­mat­i­cally in the wee hours of the morn­ing and all the data is processed and ready to be looked at by the time I roll out of bed.  

Report­ing

There are a few things to keep in mind when you report on this kind of data, since it does dif­fer from stan­dard trans­ac­tional data.  First and fore­most, you must always keep gran­u­lar­ity in mind.

For an exam­ple, above, is the num­ber of Active Cam­paigns that one of the Omni­ture mar­ket­ing teams is run­ning for the month of Octo­ber (to see the results of some Omni­ture cam­paigns, you can play this game).  Keep in mind the state-nature of the report, and you can see that they’ve got around 100 cam­paigns run­ning each day with some growth in the lat­ter end of the month.  If I look at this on a weekly basis then it would tell me that there are around 700, which is obvi­ously not cor­rect (in addi­tion to the mis­lead­ing par­tial week).

  

Monthly suf­fers the same prob­lem, but to the tune of 30k Active Cam­paigns.  In fact, any time period greater than a day dra­mat­i­cally inflates the num­bers.  If this is all we had, our analy­sis poten­tial would be rather lim­ited, but not to worry.  To the res­cue comes the Day Counter event that I men­tioned a moment ago.  By cre­at­ing a cal­cu­lated met­ric that divides Active Cam­paigns by Total Day Coun­ters (700 / 7 or 31,000 / 31), we get an Avg Active Cam­paigns met­ric that can be freely applied to any time frame.

Look­ing at the same report with my Avg met­ric yields the following:

  

Much bet­ter.  Much eas­ier to see the over­all trend here with the growth dis­played nicely (and accu­rately).  The num­ber of Active Cam­paigns this team is run­ning is increas­ing over time.  The fact that they are run­ning more and more cam­paign indi­cates to me that they are real­iz­ing the value of the prod­uct and are try­ing dif­fer­ent tests to increase their ROI.  My hope is that as they com­plete one test they get some inter­est­ing results which dri­ves one or more addi­tional tests to increase learn­ings about their audi­ence or to tar­get par­tic­u­lar con­tent to a seg­ment of their audi­ence
.

If you have any ques­tions about how state-based mea­sure­ment could apply to your busi­ness or need some help inter­pret­ing the results, feel free to email: ben / dot / robi­son / at / adobe / dot / com.  You can also find me on Twit­ter (http://​twit​ter​.com/​b​e​n​_​rob).

0 comments