A/B testing mystery (something special from CXL)
As my journey with CXL progresses, the harder it gets (and we’re only in the second week!).
In this article, I want to focus on the second module of the Growth Marketing mini degree by CXL Institute, in particular on the most difficult, curious, and mysterious topic: A/B testing.
About me
Hello, my name is Michele Rosa, I’m from Italy, and I’m a young growth marketer, who is interested in innovation, marketing, and technology!
About this series of reviews
This series of reviews (12 articles) will be focused on the Growth Marketing Minidegree of the CXL Institute, where I will write about my opinions about the mini degree and my weekly learning.
I learned about the CXL Institute through a search on Google and when I came across the number of well-qualified professionals that studied and teach there, I was very excited to start my studies there too!
I’m super excited to be sharing my learnings, and this week, I had the privilege to learn from two men I’ve admired so much through their works and writings. They are Peep Laja and Ton Wesseling.
About CXL Institute
If you’re zapping within the Digital Marketing world for a while, chances are that you’ve heard of CXL Institute. They offer marketing training courses taught by the top industry practitioners in CRO & UX, Analytics, and Marketing. Their in-depth programs are true gold for those who’ve grown tired of poor quality marketing content out there.
Successful businesses work with CXL Agency to identify growth opportunities, understand their customers, and keep increasing their revenue, thanks to the different services they offer, such as Revenue optimization, Customer research, Analytics Implementation, and Digital Intelligence.
But let’s not get lost in small talk, and go back to talking about the mini-degree.
So, this was the second week of the program and I want to explain to you what I learned.
Ready? Let’s start.
What is A/B testing
A/B testing, or ‘split-testing,’ is a method of testing variations of web pages, app interfaces, ads, or emails. Designers or marketers start with a goal of, say, earning more sign-ups and test two versions with similar audiences. If there’s a clear winner, they adopt that version. As you will see, A/B testing is critical for designing products people love.
Some benefits of A/B testing concern performance optimization, hidden trends and user behaviors detection, and even user feedback incorporation.
Our super coach Ton Wesseling list 3 moments on when you should A/B test, and I want to reveal them to you below:
- Deployments
- Research
- Optimization
Research to get insights
This was a really important topic of the master on A/B testing and in this part Ton offers the internal research model CXL use to come up with a proper hypothesis for your A/B testing.
In order to describe the model, Ton makes a comparison with the scientific method.
In science, this is what they use.
At the top they make observations, and then they think of interesting questions, in order to formulate hypotheses.
Then they develop testable predictions and gather data to refine or reject hypotheses. And repeat.
Ton describes A/B testing on the business side as exactly the same. And I want to share with you the secret model CXL use for their A/B test.
They created this fantastic model: FACT&ACT.
Although it sounds like a movie title, this template can help you run A/B test already from tomorrow.
I’m going to explain to you this model more in detail:
- We find something that needs to be solved
- We analyze this thoroughly
- We come up with hypotheses and we create the challenge here
- We test
- We analyze
- We combine results with other experiments
- And we tell our company what we’ve learned
Obviously, it’s not about one A/B test, but it refers to a whole program of A/B testing.
And if you look at the whole picture, the model looks like this:
- Studying the customer journey and learning about it takes time
- Hypotheses take less time
- But testing is much faster
What does “mystery” have to do with A/B testing
I wanted to change the title of the course “A/B test mastery” with “mystery”, in order to write this article.
Why did I make this decision?
If we focus on the first part of the FACT&ACT model, the main question about the “Find” and “Analyze” sections is “What could be optimized?”
As you know anyone has an opinion, even experts, but opinions are affected by bias and have a low quality of evidence.
So, before starting every A/B testing, you should do customer behavior researches. In fact…
without an analyzed reason to start it doesn’t make any sense to start at all.
The goals of a customer behavior study are:
- Insights into the most important customer journeys
- Understanding the basic user behavior
- Input for setting hypothesis
And the 6V Conversion Canvas (value, versus, view, validated, verified, voice) can help us through this research.
I’m not going to explain much more to you, because I do not want this article to be an extract from the course, but I would stimulate your critical thinking and help you, through my personal review and my personal thought of this mini degree.
The mini degree goes on to talk about statistics for A/B testing, which is a topic that most marketers neglect, and data analytics.
Watch out for traps
When it comes to testing it’s easy to fall into statistics traps, and one of the biggest is stopping too early. And this has to do with a couple of concepts called regression to the mean and sampling error.
The sampling error occurs when we don’t know the true conversion rate and we’re sampling from the bigger population. And in the beginning, when we start testing, we might be sampling from an obscure group, so it’s something that is not centered around the true mean. So, after a long time happens that we get regression to the mean.
For this reason, in the beginning, you may say that one is performing very poor versus another (or reverse), and you might want to call the test, stopping too early.
Sorry man, don’t do that!
Conclusion
The content is incredibly thorough and in-depth, and the value of the course highly compensates for the rushed pace. The lessons are highly practical and applicable in the real world, as the instructors are top-notch leaders in their respective fields, having a lot of real-life examples to share. The curriculum looks extremely diverse and I’m looking forward to diving deep into areas I may not be an expert in.
As a way to stay accountable, weekly, I’ll be sharing key takeaways from the program here on my blog on Medium.
