Chandal Nolasco Da SilvaYears ago our team built self-improving machine technology that could achieve bid and budget management faster and better than any human, using more data than one person could ever consider. The technology was created by rocket scientists with other early AI patents and includes more than 30 sophisticated algorithms. Not just an automated, rule-based system, this one actually self-improves constantly, making it the closest thing we have in the SEM industry to artificial intelligence.
We get asked a lot about our machine learning technology known as Acquisio TuringTM and what type of results marketers can expect from it, because agencies and marketers are sick of claims from martech providers. So our team set out to put our machine to the test.
We wanted to understand exactly what type of campaign results marketers could expect from Acquisio Turing versus accounts that were not on this technology. Jaws will drop, data will fly, heads will roll… you ready to get your hands dirty with some machine learning data? Here we go!
Parameters of the Study
The actual study included a total of 32,858 accounts between September 1, 2015 and April 1, 2017, some of which were on Acquisio Turing and others that weren’t comparing results from month 1 to month 3 from their activation date
As seen below in Lessons 1 through 4, we chose to report on both averages and medians because very large gains can dominate the average, skewing expectations. For example, if one of our clients had a 3000% increase in conversion while the rest were between 50 and 100% the average would be dragged up higher due to the outlier. The median is less susceptible to the impact of outliers.
For Lessons 5 and 6 we looked at all the accounts on Acquisio Turing as well as those that weren’t and broke comparisons down by publisher networks, either AdWords or Bing.
Lesson #1: Accounts Are Both Apples AND Oranges
Not all campaigns are created equally. The success of a campaign depends on so many factors. One campaign could have a high budget, another could have a shorter time frame, one could sell products that are more popular, while another has little competition. In order to make sense of anything and get statistically significant data, account comparisons should only be made between similar accounts.
We had to slice the data properly and compare apples with apples. Since Acquisio Turing is a bid and budget management tool, it was extremely important to compare accounts with similar budget spend when measuring the cost per click, click through rate, cost per acquisition and conversions. To determine these metrics, we therefore filtered the accounts for similar spend, within a plus or minus 10% range, leaving 8,235* accounts.
*The outliers were removed prior to calculating the averages by treating the scores as log-normally distributed and using Median Absolute Deviation method.
Lesson #2: Reduced Cost Per Click (CPC)
People turn to Google to search for services. Google allowed advertisers into search results and charges them per lead. Cost per click is the metric Google created that determines what advertisers will pay each time a person clicks on their ad. Marketers rejoice that the cost of their search advertising campaigns only goes up when campaigns are effective; however, despite the quality of the clicks there’s a cost for those leads.
For the 8,235 accounts that were comparable, we observed a 7% reduction in CPC on average between the first and third month. The median for the group was a decrease of 10%, which means that half of the accounts on Acquisio Turing had a CPC decrease of 10% or better. Overall about two-thirds saw a reduction in CPC.
Hells to the yeah for lower CPCs!!
Lesson #3: Increase in Clicks
While clicks are not the only thing that matter given that they may not convert for many reasons, we all want quality clicks – real traffic to our landing pages from interested prospects.
We saw that on average the number of clicks increased by 15% during the first three months. Again, the median for the click change between month one and month three was 8%, which means that half of the accounts on Acquisio Turing had an increase in clicks of 8% or better. Overall 59% saw an increase in clicks.
Thank you machine learning for bringing in the clicks! Time to do some converting…
Lesson #4: Cost Per Acquisition (CPA) Decrease AND Increase in Conversions
CPA is the amount advertisers pay per conversion. Conversions are the ultimate goal of any PPC campaign and of advertising in general. However, conversions can sometimes be tricky to track. From UTMs to tag manager and even third party software, things can get messy really quickly.
Of the 8,235 accounts that had a budget spend within 10% of one another, only 2,490* were tracking conversions, which means that for the conversion portion of our study we are comparing just under 2500 PPC accounts.
Of the accounts tracking conversions, the median CPA change was a decrease of 18% or better. That means half or more of the accounts reduced their cost per acquisition by 18% or better. Overall 64% of the group saw a reduction in CPC.
Of the accounts who were tracking conversions, we observed an increase in the number of conversions by 71%…which had our team all like:
However, to be conservative we should always look at the median conversion change which was a 22% increase in conversions between month one and month three. That means that half of the group improved conversions by at least 22% or better. Overall 62% of the accounts using machine learning saw an increase in the number of conversions.
Now that’s something to write home about!
*The outliers were removed prior to calculating the averages by treating the scores as log-normally distributed and using Median Absolute Deviation method.
Lesson #5: Budget Attainment Pretty Much Every Time
Budget attainment isn’t always thought of as a key metric. Recently we wrote a post on our blog about why budget attainment should be a KPI:
“If the PPC marketer overspends the budget, it’s a problem for obvious reasons. Even if overspending meant achieving another important KPI like conversions, the client simply may not have the extra money; hence, allocating a budget in the first place. If the PPC marketer underspends the budget, the client will ask why they didn’t put all the resources they were given into getting maximum results. Meanwhile no one can consistently measure a return on investment if the investment is different every month, and therefore there’s also data integrity at risk. Ultimately, if PPC marketers can’t spend budget accurately and consistently the client will want to spend their money with someone who can.”
With this rationale in mind it is extremely important for PPC marketers to attain their budget month after month. We wanted to see if machine learning could help them do that. To answer our budget attainment question we compared our accounts that were using our machine learning technology against those who weren’t. We also had to consider those that were using it for campaigns on AdWords and those who were running Bing campaigns. We looked at a total of 32, 858 accounts:
- 12, 651 were using machine learning on AdWords
- 11,094 were not using machine learning on AdWords
- 6,342 were using machine learning on Bing
- 2,771 were not using machine learning on Bing
Average Budget Attainment
For AdWords we found that accounts were on average 3.4 times more likely to pace and spend their budget as intended than accounts not using Acquisio Turing.
For Bing we found that accounts were on average 11 times more likely to pace and spend their monthly budget using Acquisio Turing than those who were not.
If we break the data down by budget spend we found the following:
- Accounts that spent less than $500 per month were 3.1 times more likely to attain their budget on AdWords and 11.3 times more likely on Bing, than accounts not using machine learning.
- Accounts that spent between $500 and $1500 per month were 2.3 times more likely to attain their budget on AdWords and 10.1 times more likely on Bing than accounts not using machine learning.
- Accounts that spent more than $1500 per month were 5.2 times more likely to attain their budget on AdWords and 18.6 times more likely on Bing, than accounts not using machine learning.
Lesson #6: The Average Lifetime Value (LTV) of Accounts Increases
The amount of time that an account lives on the platform can mean a few good things. First of all, successful campaigns are more likely to continue than those that are not performing well and get paused or nixed. Second for an agency, reseller or channel partner, that means more money. Depending on the amount of accounts this longer lifetime value represents, it can provide significantly more annual revenue at scale.
To determine what happens to the LTV of the 32,858 accounts we broke the them down by those using machine learning technology and those who weren’t. We found that those using machine learning technology lived one month longer on AdWords and two and a half months longer on Bing than those that weren’t .
What Machine Learning Can Teach You About PPC
Since the machine learning technology we applied to this study is constantly self-improving, literally getting smarter everyday, we expect that the results presented above will only get better.
TLDR Summary:
- In order to make sense of anything and get statistically significant data, account comparisons should only be made between similar accounts.
- Half of the accounts using machine learning had a CPC decrease of 10% or better. Overall about two-thirds saw a reduction in CPC.
- Half of the accounts using machine learning had an increase in clicks of 8% or better. Overall 59% saw an increase in clicks.
- Half or more of the accounts reduced their cost per acquisition by 18% or better. Overall 64% of the group saw a reduction in CPC.
- Of the accounts who were tracking conversions, we observed an increase in the number of conversions by 71%. Overall 62% of the group saw an increase in the number of conversions.
- For AdWords we found that accounts were on average 3 times more likely to pace and spend their budget as intended than accounts not using machine learning.
- For Bing we found that accounts were on average 11 times more likely to spend their monthly budget using machine learning than those who were not.
- The accounts using machine learning technology lived four months longer than those that weren’t.
From lower CPCs to higher conversion rates, longer LTVs and more, Acquisio Turing has already provided tremendous value to the accounts it runs on over the last two years. We’re really excited to share the good news with marketers like you as an increasing amount of machine learning solutions begin to shape our lives and now our SEM campaigns!
Image Credits
Feature Image: Unsplash/Maxime Bhm
All screenshots by Chandal Nolasco da Silva. Taken Summer-Winter 2017 from the latest Acquisio Turing Performance Report.
Vivien Leung
Tom Whatley