What is A/B Testing? Very simply, it is a means of testing different creatives – copy, images, calls to action – by trying different options and then going with the one that gives you the best results. It has become a widely accepted tool for digital marketers.

There are, of course, no real dangers inherent in A/B Testing – it’s a great technique for trying out different creative approaches – but, as with all statistics, we have to be careful about how we read the results.

A digital consultant recently told me he’d been doing some A/B Testing on email subject lines and had come up with some definitive results. With a test sample of 4,000 email addresses, he’d got a 9% open rate for half of the emails and 11% for the other half, so he’d sent the remainder of his list the newsletter with the subject line that gave him the higher response rate. But which is the higher number, 9 or 11? It might seem obvious but is less so if we apply variation theory and a margin of error to them.

All processes, and especially those that involve human behaviour, are subject to variability and it takes 5-6 tests to establish the level of natural variation (some would say 3-4 data points are sufficient to detect a trend). If you run a test five times and the results are 4%, 20%, 7%, 26% and 5%, then the variation is very wide and it makes it very difficult to predict what the next data point will be. If the results are much closer together you are on safer ground (it’s easier to feel secure in predicting that the next result will be between 5% and 8% than if it were between 4% and 26%).

The consultant who detected a difference between 9% and 11% on the basis of one test was in danger of inferring a false response rate for the remainder of his emails, in that he had no real clue as to what he could expect that result to be. Also, if he’d applied a margin of error to the figures, he would have seen that they might actually be the same (in the case of a smaller sample, they might have actually been reversed). 9% and 11% might be a massive difference if applied to precision-made tools but is too close to call for a measure as qualitative as responses to email subject lines.

In this instance, the difference in data was not statistically significant. If the results had been 10% and 20% it would be a no-brainer but at 9% and 11%, you could easily choose the wrong path. It might be neither here nor there whether a few more people do or don’t open an email but, in other circumstances, the lack of rigour in assessing the test could have more serious consequences.

I hear all the time how one approach to subject lines is better than all the rest and I always wonder how rigorous the testing was. Just trying it once and concluding that the result has universal application is a real danger. Coming to a potentially false conclusion and then passing it on as wisdom to your clients is even more serious.

Go ahead and test, yes, but read the results carefully and be aware of the dangers of not understanding the data you get.

1. Blogged intermittently
2. Completed a CAM Diploma in Digital Marketing
3. Learned a lot about Digital Marketing and myself
4. Decided to do a Web Analytics and Social Media Monitoring module next year
5. Completed my first year of CPD with the CIM
6. Launched a new website
7. Stopped tweeting (except for business)
8. Not used Facebook (except for business)
9. Used Twitter lots for business
10. Used Facebook lots for business
11. Stopped writing opera and theatre reviews
12. Got an apptastic iphone
13. Run lots of Facebook and Google Ad campaigns
14. Filmed a number of videos none of which have gone viral
15. Struggled with an email database that’s going downhill fast
16. Not sorted my Youtube channel out (on the list)
17. Eaten approximately 700 bananas
18. Got hooked on Made in Chelsea (oh yes)
19. Had shockingly bad experiences with Groupon and Wowcher
20. Sold about 40,000 tickets

The website I’ve been working on for the last few months has just gone live. There are, no doubt, some things to iron out over the next few weeks but I think it’s looking good. I’ve very much enjoyed working with Hans de Kretser Associates on it. They’ve done a great job and been very patient with us. I’ll be doing PPC to drive traffic and a big audience feedback exercise in the next couple of weeks.

It’s a further illustration of how easily bad publicity is communicated in this age. People will display their prejudices, without any pause for thought or weighing up of the arguments. Jewish music commentator Norman Lebrecht on his blog says that the LPO had no choice. Debatable but maybe it’s a slightly more considered response than some.

Not that I’m in any way endorsing the LPO’s actions. I’m largely pro-Palestinian (and I wasn’t outraged, like some, by the protests at the Israel Philharmonic’s prom) but there are different sides to the argument that need assessing objectively, without resorting to knee-jerk reactions. So, I’m just commenting on the medium, rather than getting involved with the debate. Email, Twitter, Facebook – all these channels promote immediate, easy responses; it’s the downside of the democratisation of opinion. It’s too easy to respond without due consideration. Maybe the LPO have a case to answer; an in-depth discussion is needed to be sure. Whether they have or not, their marketing department has a bit of a headache to deal with in the meantime.

It seems a bit daft (lazy journalism?) to suggest that the new venture is in trouble so very soon (“Has the novelty of Google+ worn off or is this just a blip on the road to world domination?” asks the editorial) and comparing just two data points, which makes no allowance for statistical variation, is always a dangerous thing to do. But, perhaps, one would expect something as big as this to be growing exponentially at this early stage and an early downturn is some kind of indicator.

Google+ debuted on 28 June and it was reported it had hit 20 million users almost immediately, with some people deserting their Facebook profiles for the new platform. Sounds like the appeal of novelty and the capacity of the public to flit onto the next big thing to me. People must be getting bored with Facebook and Twitter by now (I know I am – thoroughly bored) and something new and sparkly is bound to get people excited initially.

I’ve always found Facebook clunky and a bit frustrating (I only use it for work purposes, not personal these days) and have been waiting for a rival to do the job better and, if anyone can, Google ought to be able to. We’ll just have to wait and see where things go from here. Everyone will be watching Google+’s progress like hawks.

Anyone got any experience of Google+?

Putting aside the rights and wrongs of the argument, it’s an illustration of how uncontrollable social media can be when people scent a scandal. Opera North put a press statement out (too late for some people) and then another to try and ameliorate the effects of the first (a further statement was also issued later in the day in an attempt to contain the damage).

One day you’re happily tweeting inanities to your followers, the next you’re fending off a firestorm of (sometimes quite vicious) criticism. It’s great to be trending nationally but not when you’re being accused of everything evil under the sun. It’ll no doubt blow over but a stain is bound to linger for some time to come.

There’s an interesting response to the Twitter Storm here

At the centre is the great success of the promotional video, which has now had well over 1 million hits on Youtube. But does viral marketing work? It depends what the business objective was behind the activity. Was it to amuse a million people with a jokey video, in which case, Can I be your friend? has fulfilled its objective perfectly, it would seem. But is that a good use of taxpayers money? Hm.

Could the business objective be to sell some tickets? You’d think. However, ENO have resorted to papering the show heavily and I hear reports that it’s still playing to half-empty (half-full?) houses. I pointed out in my earlier post that only 1% of the Youtube viewers had to bring a friend to ensure a sold-out run. It would appear that hasn’t happened.

However, we don’t know if, say, 0.5% of the Youtube viewers have bought tickets, accounting for most of the sales to date (although my instinct tells me not). If that were the case, the response rate would be somewhere around the level of an average Adwords campaign. Whatever the situation, the ENO marketing team should have a good calculation of the ROI for both the video and the campaign as a whole.

Laptopera (photo: Richard Hubert Smith)

The objective could have been to raise awareness of English National Opera, which placing the name and logo before a million people would certainly do. Without some recall research, we can’t know whether any of the million actually clocked what was being presented to them and by whom.

I wrote way back about the danger of only looking at the big numbers (First Impressions). It could be that with this initiative ENO have had a major success or an alarming lack of it. Only they will know because they know what objective they set in the first place and one assumes they had rigorous measures in place to assess the success of the campaign.

Having seen Two Boys now, I still think there is something subtly clever about the video. While it doesn’t in any way portray what goes on onstage, it does play with certain ideas inherent in the show. In fact, in my humble view it does it a lot more successfully than the opera itself. The 3 minute video makes me think about some of the issues which the performance did not.

Its aim is to promote the new opera by Nico Muhly and Craig Lucas, Two Boys, which opens at the Coliseum this week (I’ll be reviewing it for Whatsonstage.com). The opera is based on a real-life story of internet connections gone wrong, when one boy stabbed another he’d met online. Dark stuff and highly relevant but the video takes a much lighter-hearted route. It would hardly go viral if it didn’t.

The video is clever not on many levels but perhaps on one or two. It is quite amusing, employing the sort of candid camera, confuse-the-public-on-the-street, technique that always entertains. A presenter goes up to people and says “Would you be my friend?” and “Could I poke you?” It’s bound to freak people out. He goes further and asks people about their use of Twitter and then says he’ll follow them.

Watch the video here

Where it gets a bit clever, and reflects some of the themes of the opera (such as I can gather without having seen it), is that it shows behaviour that people find completely acceptable online as downright weird when carried out in the street (“following” someone online is one thing, physically doing it in Oxford Street is altogether creepier).

The end of the video neatly relates what you’ve seen to the work being promoted and I must say I actually LOL’d at it (almost ROFL’d in fact). Check out the “Status Update.”

The question is will it sell any tickets? ENO need only 2% of the people who have watched the video to buy a ticket and the run will be sold out (1% if they each bring a friend). A quick glance at the booking plans for Two Boys (www.eno.org) show that there’s currently a long way to go.

The marketing team will no doubt hide behind the “raising awareness” argument (don’t we all?) but it would be really interesting if they could do a research exercise and establish the recall rate for the film. How many people remember, or even noticed, what the video is promoting?

Most of ENO’s prolific output of show-specific videos is straightforward footage of rehearsals, interviews with the director/singers etc. They generally get between 1,000 and 2,000 hits on Youtube. They must be pretty pleased with themselves at the success of Can I be your friend?

Putting your brand before a million people is a great thing (and well done to the ENO team) but how many of the viewers would actually think of visiting the venue? It’ll also be interesting to see if they can repeat the success of this one, as getting something to go viral is very hit and miss.

I’ve created a special landing page for the website, with a call to action to phone a number and then I’m using Google Adwords to drive the traffic. So, it’ll be completely trackable, down to the cost of getting the lead and then any subsequent conversion.

I’ve checked on Google’s Keyword Tool that business people are googling the appropriate keywords (which they are in sufficient numbers). The joy of Adwords is that you can schedule the ads to run when you want, so I’ve set it for weekdays 9am-5pm when people are likely to be searching and when the phone is manned.

Worth committing a bit of spend to see if this alternative means of generating leads can work.