In the last two weeks I have taken my car to the shop twice. The first visit was to get the oil changed, and just yes­ter­day I had new tires put on. Hav­ing paid for both of these main­te­nance items so close together, I can say that get­ting new tires was much more grat­i­fy­ing than hav­ing the oil changed. After the oil was changed, I noticed absolutely noth­ing dif­fer­ent about the car. A tech­ni­cian would tell you that I pro­longed the life of the car by 6 months, but I can’t see any of that ben­e­fit today. How­ever, chang­ing the tires has instantly given be bet­ter fuel econ­omy and my car no longer pulls pre­car­i­ously left into on-coming traffic.

Some busi­ness mod­els are more like chang­ing tires in that they nat­u­rally lend them­selves well to test­ing because their mon­e­ti­za­tion is so read­ily trans­par­ent. If you are sell­ing wid­gets, you can eas­ily see which test recipe sells more wid­gets. Of course if you can do that, you will likely take it a step fur­ther, look­ing at rev­enue per vis­i­tor to see which recipe hits the price elas­tic­ity sweet spot of con­ver­sion rate and aver­age order value.

If your busi­ness is media though, test­ing can feel like chang­ing oil because imme­di­ate vis­i­ble results may be dif­fi­cult to spot – even if they are there. All pub­lish­ers gen­er­ate rev­enue through their dig­i­tal prop­er­ties in one fash­ion or another. Often times this hap­pens through brand adver­tis­ing ad sales, often in the form of an impres­sion CPM model. When it comes to test­ing changes on these sites, few pub­lish­ers can make any more than a trans­par­ent con­nec­tion between what a vis­i­tor does on their site and how the income state­ment is impacted in the short term. The impact may be there, but it is rarely as read­ily appar­ent as with other busi­ness mod­els. Because of this, some mar­keters in our indus­try sim­ply resign them­selves to the fate that media test­ing just can’t be mon­e­tized. When that hap­pens, a lit­tle part of me dies inside.

Although mon­e­tiz­ing the impact of test­ing is inher­ently more dif­fi­cult for pub­lish­ers than it may be for retail­ers with a read­ily trans­par­ent mon­e­ti­za­tion model, there are still a num­ber of meth­ods to mon­e­tize the impact of your test­ing in a way that you can act off the results. Below are a few crit­i­cal points on how to do this.

One Size Does NOT Fit All
Years ago I worked for a com­pany that would have its annual con­fer­ence in a glam­orous loca­tion each year. As part of this annual pep rally, atten­dees were given com­mem­o­ra­tive t-shirts that rein­forced the company’s newest mar­ket­ing slo­gan. In an attempt to make sure that no one received a shirt that was too small, every­one received an XXL shirt. While they suc­cess­fully avoided offend­ing any­one by giv­ing them a shirt that was too small, they vast major­ity of peo­ple received shirts that were too big.  Con­se­quently, almost no one wore them.

With your media test­ing pro­gram you’re not giv­ing away shirts in bulk. You have the lux­ury of tai­lor­ing your test­ing solu­tion and met­ric to your pro­gram. Below I have out­lined a cou­ple of media test met­rics from the most sim­ple and out of the box, to met­rics that are more sophis­ti­cated and cus­tomized. This is just a sam­pling as an entire post should prob­a­bly be ded­i­cated to this list alone. What works for your orga­ni­za­tion ulti­mately depends on your organization.

Page Views Per Visit
This “engage­ment met­ric” is one of the stan­dard met­rics built into the Adobe Test&Target plat­form. It requires that you deploy a con­ver­sion mbox on every page of the site. Most orga­ni­za­tions that uti­lize this met­ric deploy this mbox glob­ally through their CMS, which can be fairly sim­ple. From there, Test&Target sim­ply counts how many pages each vis­i­tor con­sumes per visit after they have seen the test expe­ri­ence. Fur­ther­more, Test&Target keeps track of which expe­ri­ence the vis­i­tor saw and by sum­ming up the page con­sump­tion for all vis­its in a test, it can dis­tin­guish which test expe­ri­ence pro­duces more page con­sump­tion. This is a fairly sim­ply approach and it assumes that all pages carry the same mon­e­tized value, an assump­tion that is true for a select few, but not for most.

Page Score Per Visit
Many orga­ni­za­tions rec­og­nize that not all con­tent is mon­e­tized at the same rate, and for that rea­son want to weight pages dif­fer­ently when eval­u­at­ing test expe­ri­ences. This can be accom­plished by adding a page score para­me­ter to the mbox on each page of the site. With this approach Test&Target sums each of these page score val­ues for every page in a visit to pro­duce a weighted score for that visit. This method obvi­ously requires more effort to pro­duce unique page scores on every page, but the result is a met­ric more finely tuned to how rev­enue is actu­ally gen­er­ated on your site.

Few orga­ni­za­tions make it this far, but those who do often go fur­ther to tackle even more sophis­ti­cated met­rics such as ad count per visit (account­ing for pages with mul­ti­ple ads) or even ad mar­gin per visit.

Focus on Pro­gres­sion, Not Per­fec­tion
Look­ing at the options above, it is nat­ural for an orga­ni­za­tion to set its sights on set­tling for noth­ing less than a met­ric that com­pletely repli­cates their site mon­e­ti­za­tion. The prob­lem with this line of think­ing is many test­ing pro­grams stall out while work­ing on per­fec­tion when instead they could be mak­ing small, incre­men­tal steps for­ward. Your ulti­mate goal may be to become Mr. Uni­verse. How­ever, If you can’t do 10 pushups today, you need to start there.

Don’t focus on what your per­fect test­ing mon­e­ti­za­tion model is going to look like. I have seen sev­eral test­ing pro­grams fail to launch a sin­gle test ever because they were immo­bi­lized by the quest for per­fec­tion.  Instead, iden­tify where you are today and then fig­ure out what your next step looks like. With this approach you are much more likely to make progress and find some action­able test wins than if you wait for that per­fect met­ric to be ready.

Test­ing for media is always going to be a lit­tle less clear cut that with other indus­tries. Don’t give up though. Start now by iden­ti­fy­ing where you are today and deter­min­ing where you can real­is­ti­cally be tomor­row. With some strate­gic plan­ning and care­fully tuned met­rics, your test results can move beyond minis­cule oil-change increases in per­for­mance, to give your site a full engine overhaul.