I know the conventional wisdom is to set the Ads to rotate evenly and then wait till statistical significance is reached.

However, if you’re interested in increasing CTR only (conversions is not an issue), would it not also make sense to set the Ads to ‘optimize for clicks’ and let google do the math? Then replace the losing Ad with another one.

One problem I see with this is if google doesn’t give the new Ad a fair chance to rotate against the established Ad, and hence we end up with the first winner always winning. But if this is not the case, wouldn’t this be a viable Ad testing strategy?

If you go for CTR only, there is nothing better than Google’s own ‘optimize for clicks’. I don’t want to bore you with the details of why calculating statistical significance is the wrong way to go…

Just think of it this way: Google is extremely interested in your ads getting a high CTR. To Google CTR and bids are equally important. If you want to raise your CTR, Google is on your side 100%. Plus, this is a math problem. Trust their algorithms with it.

By the way, theory aside: My experience is the first winner doesn’t always win. If a better ad comes along, Google will go for it.

Excellent, thanks Martin! This makes some Ad testing much easier.

By the way, I like to understand the finer details of all things adwords, so if you ever have the time to explain (or give references to) why statistical significance (SS) is not the right way to go, I’ll be very interested. I rely on SS calculations (using z-score) to evaluate some ‘even rotation’ Ads. So I’d definitely be keen to figure out what I’m messing up!

Thanks

Hey,
Glad to help

About statistical significance & AdWords A/B testing: I plan to blog about the several aspects over at http://www.ppc-epiphany.com in about two weeks (emphasis on “plan” I’ll let you know when it’s out.

Thanks I added your blog to my rss reader and will keep an eye on it.

I’ve already started using CTR optimisation as the basis of some Ad testing. I use the number of Ad impressions as a guide as to which Ad the algorithm is favouring. However, I’m not sure at which point one can be certain of a ‘winning’ Ad. Here is an example Adgroup I have e.g.

Ad A: 65 impressions, 7 clicks (CTR = 10%)
Ad B: 112 impressions, 9 clicks (CTR = 8%)

Can one conclude that the algorithm is favouring Ad B because it got more impressions (even though in theory Ad A has higher CTR)?

Martin – that’s a great post at http://www.ppc-epiphany.com/2012/01/17/would-google-miss-your-advertising-dollars/ – really enjoyed it.

At that level of impression – Google doesn’t make a lot of assumption yet. Generally, you need more data before they really start favoring one ad over the other. Is this for a national ad or a local one? For nationals, that difference in impressions can be due to the ad serving across server farms.

That’s usually not quite a big of a deal for local ads; but its hard to draw conclusions from that few impressions/clicks.

Thanks Brad. These are national Ads in the UK. It’s a low volume market; the above figures are from the last 7 days performance.

I hope the figures will reflect an ‘obvious’ winner once they build up, and save me the need for any human intervention

Thanks Brad, great to hear that you liked it. I just created the blog a few weeks ago and I already got a lot of ideas for things to blog about from this forum…

I subscribed to your blog; looking forward to good stuff. If you want to do a couple guest posts here (with links to your blog and that you write there); just let me know (or fill out http://certifiedknowledge.org/guest-blogging/). Might help jumpstart your readership.

Wow thanks Brad. It could take few weeks, but I’ll absolutely do that. I think I have a good topic in mind…

Hi Brad,

You said earlier:
At that level of impression – Google doesn’t make a lot of assumption yet. Generally, you need more data before they really start favoring one ad over the other.

Are there any guidelines as to when one can be sure which Ad the algorithm is favoring?

I have few adgroups (with 2 Ads each) where one Ad is being shown 3-5 times as often (impressions > 500), yet the CTR of the other Ad is higher. I’m guessing the algorithm is still not sure? (this is on cpc bidding with ‘optimize for clicks’)

From a QS standpoint – Google tries to get into the 2000 impression range. I think their threshold for CTR is lower; but I really don’t know what it is. I’m guessing 500-1k impressions is when it starts to work more – but that is just a guess.

Thanks Brad