Internet Marketing Blog

New Methods for A/B Testing

July 28, 2008 – 3:01 pm

Banner ad placements, navigation, or even very specific links above or below the fold can benefit from A/B testing. You want to know if their placement is correct. You need to know that your new feature is visible. These are great reasons for A/B testing. So you write a script that sends a small percentage of your visitors to a new version of your page that you want to test. Often times every tenth visitor may be sent to the “B” page. Log files take over from there.

Although only very specific questions can be answered using this method. You’ll be able to analyze the effectiveness of the “B” page versus the “A” page often times over the course of a single day. High traffic sites can measure effectiveness of a new page design in an hour.

For example, if more people click the link you’re testing on page “B”, you can assume that more people were enticed to click the link. If page “B” drives the most traffic to your “Buy Now” button you can swap pages and enjoy the fruit of your testing labor.

Ahh, if it were just that easy… Although log files don’t lie, they strictly report the results of your testing methodology as well as any introduced variables you may or may not be aware of. There are plenty downsides to A/B testing and you need to know them.

• If a page has changed too much repeat visitors may subvert your results by not immediately adopting the new changes on the page. For this reason many sites only send new visitors to the “B” page.
• Other on-page issues could cause variables in user behavior.
• Off-page or the end-users environment could affect the results. The phone rings, it’s time for lunch when the page loads, etc. Although external factors are often the same on both “A” and “B” pages, they may not be the same at the time of visit.
• A/B testing isn’t as straight-forward as it sounds and inexperienced testers can introduce variables without noticing. A good peer review of the testing methodology could help uncover problems with the test scenario. Multiple variable require multivariate testing.
• If your test includes any sort of promotional effort that involves opinion, you’re introducing a new variable. This could be unavoidable if your testing advertisements.

There are still many key upsides to A/B testing that will prove A/B testing useful to your organization.

• With a highly capable team, multiple test scenarios can be tested quickly.
• Minimum costs compared to focus groups or in-person user testing.
• Minor changes can be tested with a high degree of accuracy in the right test scenario.
• A/B testing goes unnoticed by nearly all your test subjects.

New methods such as the Google Website Optimizer are proving extremely valuable. Take the tour and learn how to use the Google Website Optimizer to (according to Google) :

• Increase website conversion rates
• Decrease visitor bounce rates
• Increase time spent on your site
• Increase visitor satisfaction
• Eliminate guesswork from site design

We use several testing methods to determine the optimum page designs for your site. The effectiveness of an interactive marketing agency’s testing methodologies can dramatically affect the ROI on your site. We’re dedicated to maximizing your return on investment in our interactive marketing services.

Additional Reading on A/B Testing Topics

How to Split Test (A/B Test) Your AdSense Ads
http://tinyurl.com/6nny6d

A/B Testing for the Mathematically Disinclined
http://tinyurl.com/54qod2

A/B Testing in SecondLife
Announcing Linden Lab’s A/B Testing Program
http://tinyurl.com/65zpse

Putting A/B Testing in Its Place
http://www.useit.com/alertbox/20050815.html

Split A/B testing
http://www.webcredible.co.uk/user-friendly-resources/web-usability/ab-testing.shtml