A / B Testing: The Most Powerful Way to Turn Clicks Into Customers

Fun non-technical introduction to A/B testing
As someone who has been running controlled experiments (e.g. A/B tests) for over 10 years at Amazon and Microsoft, I found it fun to read with motivating examples that are easy to understand by anyone thinking of trying A/B testing.This is a book that is great to convince an org to try A/B testing, but don’t expect it to explain the technical side and the pitfalls, which you might find in peer-reviewed papers like these on exp-platform:- Controlled Experiments on the Web: Survey and Practical Guide- Seven Rules of Thumb for Web Site Experimenters- Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained- Online Controlled Experiments at Large Scale- Unexpected Results in Online Controlled ExperimentsSome things that are could be improved:- The Overall Evaluation Criterion (OEC) is not just a measure of success. This is critical and the authors’ experience with simple funnels may be limiting here. In more complicated scenarios, coming up with a good OEC that is predictive of future success is harder. For example, search engines don’t directly optimize for query share and revenue (see pitfalls paper above).- The use of A/B testing is not just for front-end features. There are great examples of back-end improvements, such as relevance for search.- One of the more important observations by groups doing A/B testing is around the importance of web site performance, which is absent here.- The authors suggest an incorrect procedure for ending experiments: “Once the test reaches statistical significance, you’ll have your answer” and “When the test has reached a statistically significant conclusion , the tester sends a follow-up with results and the key takeaways.” This procedure will have more false positives than expected. The correct procedure is to determine the duration in advance (e.g., a full week or two), then check for stat-sig at that point.

Quick Read – lots of useful data – story format is great
Well worth reading if you are in any way involved with website design, testing, and/or optimization. The book is a collection of true “stories” of experiences of the founders of Optimizely and their clients. This style of written makes the concepts and data extremely easy to take in. You’ll end up with several actionable ideas for sure!

A/B Testing is the single most important under utilized discipline in digital marketing
This is a great, easy to read guide to A/B testing . Digital marketing is custom built to test one idea against another with near instantaneous results. Read how A/B testing helped Netflix grow to become the leading online content provider and helped Barack Obama raise millions. Even better, it will give you dozens of ideas that you can use to improve your own web site. FYI, this is a book that I give clients to read…

Not really substantial. A lot of “nice” stories to fill the pages and a lot of screenshots. Obviously written to make some quick dollars. Even for those who don’t want to learn about methods: You will not learn much by reading this book.To learn about the method and its traps better read Dombrowski, Alex P. (2020): Fundamentals of A/B Testing (very good for data scientists, those who have to really do those tests).

Antes desse livro eu sabia que testes A/B eram importantes. Agora não só estou mais certo sobre isso, como aprendi a ser mais crítico e a entender que cada detalhe conta.

If you are involved in AB or multi variant testing already, I’d be surprised if you’d learn anything from this book. The content is good, although somewhat padded by success stories (with no data provided), but it is much more a high level overview of the subject. As such, I would say its suited less at the serious practitioner and more at marketing folks who want to understand this way of working. For that it is likely good.This is a serious subject, complex to really appreciate the details and its a shame this book didn’t stretch to that.