How the A/B testing and Experimentation works

Q: “What is the process for setting up a conversion test?”

A: The process is as follows:

Phase 1: Scope

We begin by assessing the structure of your website, and web stats around traffic and conversions, to identify the most valuable pages to test.

We then define what elements to test on each page – and what not to test.

Phase 2: Conversion Review

Next, we develop a plan for how each element may be improved. This becomes our internal specification for coders, copywriters and designers.

Phase 3: Test Design

Designers, copywriters and coders then work together to develop the individual elements to be tested – and import them into our testing platform as saved drafts.

Phase 4: Approval

At this point, we seek your approval on each individually tested element before the test goes live.

To minimise the need for review and revision, we value receiving brand design documentation, style guides, corporate visual identity documents, etc at the beginning of the project.

Phase 5: Monitoring and Reporting

As the test goes live, we monitor statistics closely to identify any potential issues.

We continue to monitor test statistics to provide feedback on what works, and what doesn’t, and guide any future conversion testing decisions.

However, our goal is to maximise conversions, ensuring that any winning test variations are “statistically significant to a 95% confidence interval” – meaning highly likely to continue to outperform other variations, and continue to deliver a strong improvement to your website conversions.

At this point, we cease testing that particular element – and work with you to ensure this change is made permanent on your website.

Q: “How Can We Help You To Get Better Results?”

The more insight you can provide us into your marketing – what works well, and what doesn’t, the better our work will be.

This information may include:

  • Design documentation (branding guidelines, style guides, visual identity documents, etc);
  • Website analytics including conversion and traffic statistics;
  • Photographs and stock art for use in image testing;
  • Insights (anecdotal or statistical) into what has worked well and what hasn’t in previous marketing campaigns.

Q: “What Access Do You Need To The Website’s Code Base or Backend Systems?”

A: Our split testing platform relies on a single piece of JavaScript code that is inserted once (site-wide) inside the HTML Head element on your website.

This single piece of JavaScript delivers tests to visitors (client-side) and tracks conversions – and once it is set up, no further changes need to be made to the website.

Our preference is to ask your webmaster to make this change at your end.

Q: “Does this method support secure website communications protocols (Transport Layer Security/TSL/SSL/HTTPS)?”

A: Yes. The JavaScript testing and tracking code supports both HTTP and HTTPS protocols.

This lets us test and improve any page on the website.

Q: “If you do not need to change the website, where are test variations stored?”

A: Test variations (including images) are stored externally, inside our testing platform.

This system serves tests, monitors statistics, and reports on results – and is designed to scale, combining Amazon’s CloudFront with SoftLayer services. At its current scale, it can easily handle testing for 100,000 visitors per day – a ceiling that can be raised further if needed.

Q: “Are there any security concerns in your JavaScript-based testing platform?”

A: No. It utilizes all best practises around data security, and is trusted by governments, publicly listed companies, and major banks.

Q: “What Content Management Systems (CMS) Can You Work With?”

A: Because tests are run client side, and managed through a single line of JS, the testing software that we use is compatible with all CMS platforms – including includingWordpress, Drupal, vBulletin, Magento, Shopify, Drupal, osCommerce, and custom internal solutions.