Quality Score Decoded?

Steve Baker at epiphany put up a very interesting post this week, in which he analyzed some quality score data to try and answer three questions:

1. How high is a high click through rate?
2. What is a decent click through rate for a given position?
3. How do you know if your Quality Score is being dragged down by the Account Quality Score or your adverts?

These are things we’d all like to know, and his results are interesting, but I have some concerns about whether or not they really answer any of these questions in any way we can rely on. To be clear, I’m not sure – so I’m posting my thoughts here to hopefully further the discussion. If you haven’t please go read his entire post. There three thing that concern me about the methodology and the conclusions:

1. A mistake concerning the idea that ‘quality score is only calculated on Exact Match’.
2. The assumption that ‘visible quality score’ is quality score.
3. The treatment of the relationship between quality score and bids and position.

Quality Score and Match Type

As discussed at length last week, visible quality score only takes into account the performance of past impressions where search query was identical to keyword, regardless of match type. Using a data set comprised only of Exact Match keywords is certainly a study of its own, but may very well not be representative of how all keywords of all match types perform or behave. Since AdWords already disregards non-identical queries, given the other assumptions this analysis would be equally accurate with all match types included.

Analyzing Visible Quality Score

It’s very hard not to conflate quality score and visible quality score, as Google themselves use the one name ‘quality score’ to refer to both – but they’re very different and I think as search managers we need to begin to really understand that these two things are very different and using them interchangably will lead us to a lot of very inaccurate conclusions. I wrote about the differences in a guest post last week on PPC Hero. The complexity is that if you’re only analyzing queries which are identical to keywords, as visible quality score does, when in fact all non-identical queries are earning potentially very distinct quality scores for those same keywords, then there is no way to know how valid any conclusions really are. In effect, it’s taking a non-random sampling of the available data (only the identical queries, which represents an unknown % of the data) and ignoring the rest. We might assume that the identical queries have higher CTRs and therefore represent the best quality scores of the bunch – but it is literally impossible to know. Of course, visible quality score is all we’ve got. Therefore it’s entirely natural to analyze this data and try to understand it and learn from it and draw conclusions. I’m not arguing against it. But I am suggesting that the characteristics of that data have to be acknowledged and considered along with any conclusions.

Quality Score, Bid, and Position

We all know that bid x quality score = ad rank, which determines the position in which any ad appears. In this case, quality score is not visible quality score but a version I’ve taken to calling ‘quality score for ad rank’ that includes a number of factors ignored in visible quality score. In his post, Steve supplies some very nice charts showing the relationship between position and quality score from his data set. He’s found keywords with visible quality scores of 10 that live in nearly every position from 1 down to 8, for example. Actually his post includes similar charts from many different quality scores.

Chart from Steve Baker @ epiphany

Steve’s draws several conclusions from this data:

• “It appears that Google expect the click through rate in any position to be about 65% of the next position up. So where position 1.0 has an average click through rate of 34%, position 2 has an average click through rate of 22.1%”
• “This appears to be Google’s estimate of what ‘should’ happen to your click through rate every time you drop a position – you lose just over 1/3 of your clicks.”
• “Using this, you can potentially ‘health check’ your account. If you have a click through rate of 4.5% in position 4, you should have a Quality Score of around 7 or so. If you are getting less than the predicted Quality Score across the bulk of your keywords (excluding brand, on Google only, on Exact Match), then it’s a sign that your account has other issues, possibly with the landing page, keyword relevance or the overall account quality.”

Ignoring for a moment the issues about match type and visible quality score, I just can’t quite see how these conclusions are valid. My concern is that the impact of bid on determining the position a keyword earns isn’t considered or reflected – it isn’t just quality score that is driving these positions. I’ve only had a few moments over the past 24 hours to really think about this, but I’d love to hear from the many smart readers we’ve got what they think of this analysis. It would be great to have ANY answers to the original questions, and Steve has done a great job of collecting data and presenting it to us with some interesting potential conclusions. I hope he doesn’t mind if we try to crowdsource some additional work on his data. UPDATE: I realize re-reading this that I didn’t comment directly on the three questions Steve set out to answer. They’re great questions, and there is a lot we know about the answers outside of the data being discussed here. I’ll take these up in a future post. Quality Score in High Resolution New 225-pg paperback by Craig Danuloff   Learn more and order your copy today.