Lies, Damned Lies...

Building the Perfect Display Ad Performance Dashboard, Part II – metrics

Welcome to the second installment in my Building the Perfect Display Ad Performance Dashboard series (Note to self: pick a shorter title for the next series). In the first installment, we looked at an overarching framework for thinking about ad monetization performance, comprised of a set of key measures and dimensions. In this post, we’ll drill into the first of these – the measures that you need to be looking at to understand your business.

 

How much, for how much?

As we discussed in the previous post, analysis of an online ad business needs to focus on the following:

• How much inventory was available to sell (the Supply)
• How much inventory was actually sold (the Volume Sold)
• How much the inventory was actually sold for (the Rate)

Of these, it’s the last two – the volume sold and the rate at which that volume was sold – where the buck (literally) really stops, since these two combine to deliver that magic substance, Revenue. So in this post we’ll focus on volume sold, rate and revenue as the core building-blocks of your dashboard’s metrics.

Volume, rate and revenue are inextricably linked via a fairly basic mathematical relationship:

Revenue = Rate x Volume

Another way of thinking about this is that these three measures form the vertices of a triangle:

Some business and economics textbooks call Rate and Volume “Price” and “Quantity” (or P and Q), but the terms we’re using here are more common in advertising.

Different parts of an ad business can be driven by different corners of the triangle, depending on the dynamics of how each part is transacted. Here are some examples:

• Ads sold on a time-based/”sponsorship” basis are best thought of as driving revenue performance, because deals are done on a revenue basis regardless of volume/rate (though the advertiser will have a volume & rate expectation, which they’ll want to be met).
• For premium ads sold on a CPM basis, deals revolve around Rate; the name of the game is to add value to inventory so that, impression-for-impression, it achieves more revenue.
• For remnant ads and networks, volume is king (assuming you can maintain a reasonable rate) – you’re looking to maximize the amount of inventory sold, and minimize the amount that has to be given away or sent to “house” advertising.

Because of these different dynamics, measurement of ad monetization can easily fragment into various sub-types of measure; for example, as well as cost-per-thousand (CPM) rate, some ads are purchased on a CPC or CPA basis. So a more complete version of the diagram above looks like this:

However, it’s essential to remember the key relationship and dynamic between rate, volume and revenue, which is manifested in the CPM, Impressions and Delivery Revenue measures in the diagram above. So let’s look at these measures.

 

Volume

In the online ad business, Volume is measured in Ad Impressions. I have talked about ad impressions before on this blog, in this installment of Online Advertising 101 (you may want to take a moment to read the section entitled “What’s the product?” in that post). From a measurement point of view, whenever your ad server serves an ad (or more accurately, fields a request for an ad), its measurement system should log an ad impression. How much data is logged with this impression will vary depending on the ad server you’re using, but will likely include most of the following:

• Date & time of the impression
• Advertiser
• Campaign and/or creative
• Location/placement (i.e. where the ad was served)
• Attributes of the individual who requested the ad (e.g. targeting attributes)

We’ll come back to those attributes (and how you can use them to segment your impressions for better analysis) in another post.

Capturing a true view of the ad impressions on your site can be a little more challenging if you are using multiple ad servers or networks to sell your inventory, particularly if you are using a combination of your own first-party ad server (for example, DFP) and redirecting some impressions to a third-party such as an ad network. When you have delivery systems chained together in this way, you may need to combine the impression counts (and other data) from those systems to get a true picture of impression volume, and you will need to be careful to avoid double-counting.

For reasons that will become clearer when we get on to talking about rate, it’s essential that you capture impression counts for your ad sales where you possibly can, even for parts of your site or network where the supply is not sold on an impression basis.

Other volume measures such as Clicks and Conversions become very useful when you’re looking to assess how valuable your inventory is from an advertiser perspective, since both are a proxy for true Advertiser ROI. They’re also useful for deriving effective rate, as we’ll see below.

 

Rate

At the highest level, rate is a simple function of volume and revenue – simply divide your total revenue by your total volume (and usually multiply by 1,000 to get a more usable number) and you have your overall rate – in fact, you have the most commonly used kind of rate that people talk about, known as “Effective Cost-per-Mille (Thousand)”, or eCPM (don’t as me why the e has to be small – ask e.e. cummings). Just to be clear, eCPM is calculated as:

eCPM = (Revenue) * 1000 / (Volume)

Sometimes eCPM is known as eRPM (Where the R stands for “Revenue”).

The reason we’re talking about eCPM before revenue in this post is because many advertising deals are struck on a CPM basis – i.e. the advertiser agrees to buy a certain amount of impressions at a certain pre-agreed rate. However, even for inventory is not being sold on a CPM basis, it’s essential to be able to convert the rate to eCPM. Here’s why.

The beauty about eCPM is it is the lowest common denominator – regardless of how a particular portion of your impression supply was sold (e.g. on a cost-per-click basis, or on a “share of voice” or time-based basis), if you can convert the rate back into effective CPM you can compare the performance of different subsets of your inventory on a like-for like basis. Consider the following example of delivery info for the parts of a fictional autos site:

Site area	Sold as…	Deal
Home page	Share-of-voice	$10,000 up-front
Car reviews	Reserved CPM	$2.50 CPM
Community	AdSense	$1.20 CPC

With just the information above, it’s impossible to understand whether the Home Page, Reviews or Community site areas are doing better, because they’re all sold on a different basis. But if you add impression counts (and, in the case of the Community area, click counts), it’s possible to derive an overall rate for the site, as well as to see which parts are doing best:

Site area	Sold as…	Deal	Impressions	Clicks	CPC	Revenue	eCPM
Home page	Sponsorship	$10,000 up-front	5,347,592	n/a	n/a	$10,000	$1.87
Car reviews	Reserved CPM	$2.50 CPM	3,472,183	n/a	n/a	$8,680.45	$2.50
Community	AdSense	$1.20 CPC	1,306,368	5,832	$1.20	$6,998.40	$5.36
Total			10,126,144			$25,678.85	$2.53

See? Who knew that the Community area was throwing off so much money per impression compared to the other areas?

eCPM isn’t the only rate currency you can use, though its connection to both volume and revenue puts it at a distinct advantage, and it means most to publishers because it speaks to the one thing that a publisher can exert (some) control over – the volume of impressions that are available to sell.

 

Revenue

If you sell your inventory on a fairly straightforward CPM or CPC basis, then your site’s revenue will pop neatly out of the equation:

(Revenue) = (eCPM) * (Volume) / 1000

However, if you’re running a larger site and engaging in sponsorship-type deals with advertisers, your revenue picture may look a little more complex. This is because “sponsorships” (a term which covers a multitude of sins) can contain multiple revenue components, some of which can be linked to ad delivery (and which therefore lend themselves to rate calculations), and some of which cannot.

For example, the sponsorship deal on our fictitious autos site referenced above could in fact contain the following components on the invoice sent to the advertiser or agency:

Item	Cost	Impression Target
100% Share-of-voice rotation, 300x250, Home Page (1 day)	$6,000	3,000,000
100% Share-of-voice rotation 120x600, Home Page (1 day)	$4,000	3,000,000
Sponsor branding – Home Page background (1 day)	$8,500	n/a
Sponsored article linked from Home Page (1 day)	$3,500	n/a
Sponsor watermark on Home Page featured video (1 day)	$1,500	n/a

In the above table, only the first two items are expected to be delivered through the ad server; the other three are likely to be “hard-coded” into the site’s CMS and actually deliver with the page impressions (or video stream, in the case of the last one).

There are a couple of different options for dealing with this second kind of revenue (which we’ll call “non-delivery” revenue) which can’t be directly linked to ad impressions. One is to attribute the revenue to the ad delivery anyway, kind of on the assumption that the ads “drag along” the other revenue. So in the above example, with 5,347,592 impressions delivered across the two units, the “overloaded” eCPM for the ad units would be $4.39.

The challenge with this approach is that the extra revenue is not associated with delivery of any particular ad. So in the above example, if you wanted to calculate the eCPM for just the 120x600 unit on the home page (perhaps across an entire month), would you include the non-delivery revenue? If yes, then how much of it? 50%? 40%? The lack of ability to truly associate the revenue with ad delivery makes these kinds of calls incredibly hard, and open to dispute (which is the last thing you want if you are presenting your numbers to the CEO).

The other approach is to treat the “non-delivery” revenue as a separate bucket of revenue that can’t be used in rate calculations. This keeps the data picture simpler and more consistent on the “delivery” side of the house, but you do end up with an awkward block of revenue that people are constantly poking and saying things like “I sure wish we could break that non-delivered revenue out a bit more”.

 

A complicated relationship

Once you have your arms around these three core measures, you can start to see how they interact, and there lies the magic and intricacy of the dynamics of selling display advertising. The implacable logic of the simple mathematical relationship between the three measures means that if one changes, then at least on of the others must also change. Only by looking at all three can you truly understand what is going on. We’ll dig into these relationships more in subsequent posts, but here’s a simple example of the rate achieved for ads sold on a fictional news site home page:

Someone looking at this chart may well ask “OMG! What happened to our rate in June 2009?” Well, a quick search on Wikipedia will reveal that a certain “King of Pop” died in that month, sending the traffic (and hence the ad impression volume) of most news sites sky-rocketing. In our fictional home-page example, almost all revenue is driven by “share of voice” (time-based) deals, so all that extra volume does is depress the effective rate, because the site earns the same amount per day regardless of traffic levels. So here’s volume and revenue from the same data set, to round out the picture:

We can now see that in fact, June wasn’t a bad month for Revenue; it was the huge spike in traffic that did the rate in.

The above example takes something very important for granted – namely, that we have enough segmentation (or “dimensional”) data associated with our measures to be able to break down site performance into more useful chunks (in this case, just the performance of the home page). In the next blog post, we’ll look at some of the most important of these dimensions. Stay tuned!

Should Wikipedia accept advertising?

It’s that time of year again. The nights are drawing in, snow is starting to fall in the mountains, our minds turn to thoughts of turkey and Christmas pudding, and familiar faces appear: Santa, Len and Bruno, and of course, Jimmy Wales.

If you are a user of Wikipedia (which, if you’re a user of the Internet, you almost certainly are), you’ll likely be familiar with Jimmy Wales, the founder of Wikipedia and head of the Wikimedia Foundation, the non-profit which runs the site. Each year Jimmy personally fronts a campaign to raise funds to cover the cost of running Wikipedia, which this year will amount to around $29m.

The most visible part of this campaign is the giant banner featuring Jimmy Wales’s face which appears at the top of every Wikipedia article at this time of year. This year the banner has caused some hilarity as the position of the picture of Jimmy just above the article title has provided endless comic potential (as above), but every year it becomes increasingly wearisome to have Jimmy’s mug staring out at you for around three months. Would it not be easier for all concerned if Wikipedia just carried some advertising?

Jimmy has gone on record as saying that he doesn’t believe that Wikipedia should be funded by advertising, and I understand his position. To parse/interpret his concerns, I believe he’s worried about the following:

• Accepting advertising would compromise Wikipedia’s editorial independence from commercial interests
• Ads would interfere with the user experience of Wikipedia and be intrusive
• Wikipedia contributors would not want to contribute for free to Wikipedia if they knew it was accepting advertising

I’m biased, of course, since I work for Microsoft Advertising, but I believe that each of these concerns is manageable. Let’s take them one by one:

Concern 1: Ads would compromise Wikipedia’s independence

There are plenty of historical examples where a publication has been put in a difficult position when deciding what to publish because of relationships with large advertisers. Wikipedia certainly doesn’t want, for example, Nike complaining about the content of its Wikipedia entry. And the idea of Wikipedia starting to employ sales reps to hawk its inventory is a decidedly unedifying one.

But Wikipedia does not have to engage in direct sales, or even non-blind selling, to reach its financial goals with advertising. The site could make its inventory available on a blind ad network (or ideally multiple networks) so that it would be impossible for an advertiser to specifically buy ad space on Wikipedia. If an advertiser didn’t like their ads appearing on Wikipedia, most networks offer a site-specific opt out, but the overall impact of this to Wikipedia would be minimal – Wikipedia carries such a vast range of content that it has the most highly diversified content portfolio in the world – no single advertiser could exert any real leverage over it.

Concern 2: Ads would make Wikipedia suck

As has been noted elsewhere, there are plenty of horrible ads at large in the Internet – intrusive pop-ups, or horrible creative. It would certainly be a valid concern that Wikipedia would suddenly become loaded with distracting commercial messages. But according to the back-of-an-envelope calculations I’ve done, there is no need for Wikipedia to saturate itself with ads in order to pay the bills.

According to the excellent stats.wikimedia.org site, Wikipedia served almost exactly 15bn page views world-wide in October 2011 (around half of which were in English). Assuming no growth in that figure over 12 months, that’s around 180bn PVs per year. So to meet its funding requirements, Wikipedia would need to generate a $0.16 eCPM on those page views (assuming just one ad unit per page). That’s a pretty modest rate, especially on a site with as much rich content as Wikipedia. It would give the site a number of options in terms of ad placement strategy, such as:

• Place a very low-impact, small text ad on every page
• Place a somewhat larger/more impactful ad on a percentage of pages on a rotation, and leave other pages ad free
• Place ads on certain types of pages, leaving others always ad free (such as pages about people or companies, or pages in a particular language/geo)
• Deploy a mix of units across different types of page, or in rotation

This also assumes that Wikimedia needs to raise all its funds every year from advertising, which it may not need to – though once the site accepted advertising, it would definitely become more difficult (though perhaps not impossible) to raise donations.

To preserve the user experience, I would definitely recommend just running text ads, which could be placed relatively unobtrusively. Sites running text-based contextual ads (such as those from Google AdSense or Microsoft adCenter) can usually expect to get at least around $0.30 eCPM, so there would be some headroom.

I would also recommend that Wikipedia not run targeted ads – or at least, only work with networks that do not sell user data to third parties. It could cause significant backlash if it became felt that Wikipedia was effectively selling data about its users’ browsing habits to advertisers for a fast buck.

Concern 3: Ads would make contributors flee

I can speak to this concern less authoritatively, since I am not that familiar with the world of Wikipedia contribution, but so long as Wikimedia made it clear that it was remaining a non-profit organization, and continued to operate in a thrifty fashion to cover its costs, the initial outrage of Wikipedia contributors could be managed. After all, plenty of other open-source projects that rely on unpaid contributors do provide the foundations for commercial activities, Linux being the best example.

In any case, in its deliberations about balancing the needs of its contributors with its need to pay the bills, Wikimedia will need to face some hard questions: Will it always be able to cover its costs through donations? Does the current level of investment in infrastructure represent an acceptable level of risk for a site that serves so many users? Is it acceptable to rely on unpaid contributors indefinitely? If Wikipedia ran out of cash or went down altogether, the righteous indignation of its contributors may not count for very much.

Apart from advertising and donations, the only other way that Wikipedia could pay the bills would be by creating paid-for services – for example, a research service. But would the unpaid Wikipedia contributors really be happier with this outcome than with advertising? It would effectively amount to selling the content that they’d authored for free. At least with advertising, it’s the user that is the product, not the content. So long as Wikipedia can maintain editorial independence and retain a good user experience, advertising feels like the better option to me.

Building the Perfect Display Ad Performance Dashboard, Part I – creating a measurement framework

There is no shortage of pontification available about how to measure your online marketing campaigns: how to integrate social media measurement, landing page optimization, ensuring your site has the right feng shui to deliver optimal conversions, etc. But there is very little writing about the other side of the coin: if you’re the one selling the advertising, on your site, or blog, or whatever, how do you understand and then maximize the revenue that your site earns?

As I’ve covered previously in my Online Advertising 101 series, publishers have a number of tools and techniques available to manage the price that their online ad inventory is sold for. But the use of those tools is guided by data and metrics. And it’s the generation and analysis of this data that is the focus of this series of posts.

In this series, I’ll unpack the key data components that you will need to pull together to create a dashboard that will give you meaningful, actionable information about how your site is generating money – or monetizing, to use the jargon.

We’ll start by taking a high-level look at a framework for analyzing a site’s (or network’s) monetization performance. In subsequent posts, we’ll drill into the topics that we touch on briefly here.

 

Getting the measure of the business

Ultimately, for any business, revenue (or strictly speaking, income or profit) is king. If you’re not generating revenue, you can’t pay the bills (despite what trendy start-ups will tell you). But anyone running a business needs a bit more detail to make decisions that will drive increased revenue.

In the ad-supported publishing business, these decisions fall into a couple of broad buckets:

• How to create more (or more appealing) supply of sellable advertising inventory
• How to monetize the supply more effectively – either by selling more of it, or selling it for a better price, or both

Another way of thinking about these decisions is in a supply/demand framework that is common to almost all businesses: If your product is selling like hot cakes and you can’t mint enough to meet demand, you have a supply problem, and you need to focus on creating more supply. If, on the other hand, you have a lot of unsold stock sitting around in warehouses (real or virtual), you have a demand problem, and you need to think about how to make your products more compelling, or your sales force more effective, or both.

Online publishers usually suffer from both problems at the same time: Part of their inventory supply will be in high demand, and the business will be supply-constrained (it is not easy to mint new ad impressions the way a widget manufacturer can stamp out new widgets). Other parts of the inventory, on the other hand, will be hard to shift, and the business will be demand-constrained – and unlike widgets, unsold ad inventory goes poof! when the clock strikes midnight.

So analysis of an online ad business needs to be based on the following key measures:

• How much inventory was available to sell (the Supply)
• How much inventory was actually sold (the Volume Sold)
• How much the inventory was actually sold for (the Rate)

It’s ultimately these measures (and a few others that can be derived from them) that will tell you whether you’re succeeding or failing in your efforts to monetize your site. But like any reasonably complex business (and online advertising is, at the very least, unreasonably complex), it’s really how you segment the analysis that counts in terms of making decisions.

 

What did we sell, and how did we sell it?

Most businesses would be doing a pretty poor job of analysis if they couldn’t look at business performance broken out by the products they sell. A grocery chain that didn’t know if it was selling more grapes or grape-nuts would not last very long. Online advertising is no exception – in fact, quite the opposite. Because online ad inventory can be packaged so flexibly, it’s essential to answer the question “What did we sell?” in a variety of ways, such as:

• What site areas (or sub-areas) were sold
• What audience/targeting segments were sold
• What day-parts were sold
• What ad unit sizes were sold
• What rich media types were sold

The online ad sales business also has the unusual property that the same supply can (and is) sold through multiple channels at different price points. So it is very important to segment the business based on how the supply was sold, such as:

• Direct vs indirect (e.g. via a network or exchange)
• Reserved vs remnant/discretionary

Depending on the kind of site or network you’re analyzing, different aspects of these what and how dimensions will be more important. For example, if you’re running a site with lots of high-quality editorial content, analyzing sales by content area/topic will be very important; on the other hand, if the site is a community site with lots of undifferentiated content but a loyal user base, audience segments will be more relevant.

 

Bringing it together – the framework

I don’t know about you, but since I am a visual person to start with, and have spent most of the last ten years looking at spreadsheets or data tables of one sort or another, when I think of combining the components that I’ve described above, I think of a table that looks a bit like the following:

This table is really just a visual way of remembering the differences between the measures that we’re interested in (volume, rate etc) and the dimensions that we want to break things out by (the “what” and “how” detail). If you don’t spend as much of your time talking to people about data cubes as I do, these terms may be a little unfamiliar to you, which is why I’m formally introducing them here. (As an aside, I have found that if you authoritatively bandy about terms like “dimensionality” when talking about data, you come across as very wise-sounding.)

In the next posts in this series, I shall dig into these measures and dimensions (and others) in more detail, to allow us to populate the framework above with real numbers. We’ll also be looking at how you can tune the scope of your analysis to ensure that

For now, here’s an example of the kinds of questions that you would be able to answer if you looked at premium vs non-premium ad units as the “what” dimension, and direct vs indirect as the “how” dimension:

 

As this series progresses, I’d love to know what you think of it, as well as topics that you would like me to focus on. So please make use of the comments box below.