‘Tis the Season for Experimentation!

Retail experimentation:  Do not fear the holidays!

If you’ve been naughty in Holland, St. Nicholas beats you with a switch, throws you into a sack and kidnaps you to Spain.  User Experience teams fear similar treatment if they run experiments during the holidays.

What we hear typically is: 

“IT has a code freeze until January.”

“We can’t take risks over the holidays.”  

But managing risk is a function of balancing downside and upside.  

2021 was the biggest holiday season ever according to the National Retail Federation and US Census Bureau data.  Online sales were a record $238.9 billion last year and are expected to increase 10% – 12% in 2022 (vs. 6% – 8% overall).

NOW is THE most important time to test

Start your thought process around opportunity instead of fear.  Ask yourself how valuable it would be to:

  • Understand how your most valuable audiences convert better
  • Leverage holiday traffic levels to run tests more rapidly
  • Run experiments at a time when they could generate the highest returns

Between now and the end of your holiday shopping season, look to do experiments that have these characteristics:

  • Low technical effort 
  • No process dependencies
  • Are in areas or functions that are unique to the holidays

Experiments that require a lot of code also take more time to develop and test, so focus on things like instructional language and images where it is relatively easy to test multiple options.  Line up multiple iterations so you can keep testing velocity high. A lift in traffic from holiday shoppers further mitigates risk because you can run effective tests with a smaller percentage of your site visitors.  

A 0.5% increase in conversion over the holidays is worth many times more than the same result during another time of year — so be sure and track and report the incremental revenue generated from your holiday experiments.

You will set yourself up to have a great New Year!