Search Engine Strategies (SES) Presentation Recap – Truth in SEM Analytics

At the Search Engine Strategies Conference in San Jose this week I participated on a panel entitled “Identify, Analyze, Act: SEM by the Numbers”. Below are my presentation slides, and some accompanying thoughts and comments.

Of course, it’s great to see such a high-profile discussion of PPC Analytics. I’d love to dig in and talk about very specific strategies and tactics, metrics and applications, etc. But with limited time, and with most paid search managers up against some real barriors to applying solid analytics to their campaigns, I decided to use the opportunity to talk about these challenges in the hopes that raising visibility would encourage the PPC community to start requesting the kinds of changes we need from the engines and tools vendors and the ‘best practices’ widely discussed across the paid search community in order to make meaningful analysis easier and more commonplace. Invisibility The first challenge to paid search analytics is that a lot of the data you’d want to analyze isn’t readily available. The prime example, which I’ve covered in detail before on this blog is search query information. Google Adwords and Analytics (or Yahoo or MSN) don’t provide it at a keyword level, and neither to a great many web analytics or even PPC management tools. Similarly, SKU-level margins which enable SKU-level profit to be calculated to enable ROI to replace ROAS, is not available in most tools. That has been discussed a length too. But if we’re trying to measure how people who search react to our ads in terms of our profitability, it’s rather amazing that a clear view into their search and our profit isn’t a default condition of analysis. Deception Another challenge to PPC managers is data that isn’t what it appears to be – untrustworthy data. Some of it is inaccurate, some of it is simply misleading. None of it helps. Averages are the backbone of most paid search reports – the average ROI/ROAS for a campaign, the average cost-per-click for an adgroup, the average order value for one engine vs another. But averages mask as much as they summarize. Averages without awareness of the standard deviation (or dispersion) of their data should be considered of limited value. Yet millions of PPC reports are consumed every day leaving impressions and causing decisions based on the ‘feel’ of these averages. Similarly, raw performance data is presented in all kinds of PPC reports without regard for the statistical significance or margin-of-error of that data. Snap decisions to turn of text-ads, pause keywords, or inversely let them run can be based on data which would tell a very different story if reviewed slighly later in the process. Why don’t the tools use conditional formatting or some other method to warn you that ‘there isn’t enough sample sizes to evaluate these numbers yet?’ And lastly I didn’t get the chance to speak on one last but very important point. And there isn’t enough time to go into it in detail here, but just about every revenue or profit number used to report or analyze paid search is ‘wrong’ or at least ‘suspect’ based on huge limitations in how keywords/clicks are matched with revenue over time. Everyone knows that many people visit sites a number of times before they buy, and the question of which visit and driving method gets ‘credit’ for the sale is frequently discussed. Most search and analytics programs use the last-visit standard, although a few allow a first-visit option and some even enable the choice of a linear allocation. On the SES show floor I even learned of one package that does a ‘reverse time-decay’ allocation IF the last keyword is a brand term. All of these have issues, but what’s more important is that they skew the performance of different types of keywords, and it’s hard to imagine that influence is fully considered when the eventual keyword and bidding decisions are made. (MUCH more about this later). Unlimited Power and Resources A few more challenges to taking advantage of search analytics. One is the shear size of the data we’re reviewing. Huge campaigns with hundreds of thousands of keywords, sometimes hundreds of Campaigns and Ad-Groups, and time-marches-on producing a wide range of time frames and performance trends. Plus of course the engines change, businesses have seasonality, competitors keep moving, etc. Taking this all in, keeping the factors in mind, and driving to meaningful conclusions is not easy. Within this world, and due to many of these factors, we’re making a near constant stream of changes to our campaigns. But are we making these changes carefully? When changes are made, the current crop of tools don’t help us to record the time and date of the change, watch what happens over a significant number of days/clicks/conversions, and then remind us to confirm, extend, or roll-back the change. We’re conducting tests without any test plan, test scoring, or final review. On a related note, most of the changes we’re making – adding keywords, shifting bids, modifying Match-Types, rewriting text ads – have impacts which ripple through our campaigns and certainly reduce the clarity of our reports. If we look at a two week period when 457 changes were made throughout the account in clumps throughout that time frame, what are we really learning about either what happened or what we should do next? Lessons To Learn As the above hopefully suggests, the core issues in being more analytical with paid search campaigns are not simply to produce and review more reports. We have to get clarity and accuracy from our data first, apply sensible practices to the construction and management of our campaigns, and raise the bar for both the tools we demand and use and the way we understand and use the reports they provide. We don’t need perfect solutions to any of these problems. Those aren’t coming soon and aren’t really necessary. But we should take steps to both understand and factor in these issues as we work to better learn what’s really going on within our accounts and how we can use that information to make better decisions and drive better results moving forward. We also have to understand these issues and demand that the engines and tool vendors work towards handling them in more comprehensive and reasonable ways. Both sides have ignored these issues for too long. The Match Type Keyword Trap To include at least one practical element, I included a slide summarizing the several posts that live earlier in this blog about how to target queries more specifically using Match Types. Final Thoughts The feedback after the session was great, although I probably and characteristically bit off more than the time allowed. I hope this exposition helps those who were there and those who were not. As always, I’d be happy to answer questions or discuss it further in the comments.