Ronny Kohavi

Guest: Ronny Kohavi - Former VP and Technical Fellow of Relevance at Airbnb. Ronny is recognized as a leading expert on A/B testing and experimentation, having led experimentation platforms at Microsoft and Amazon, and authored the book "Trustworthy Online Controlled Experiments." Key Takeaways: Test Everything: Every code change or feature should be tested, as even small changes can have unexpected impacts. Failure Rate: Expect a high failure rate in experiments; typically 80-92% of experiments do not improve key metrics. Overall Evaluation Criterion (OEC): Define what you are optimizing for, ensuring it aligns with long-term user value and business goals. Trust in Experiments: Building a trustworthy experimentation platform is crucial; errors can lead to significant misjudgments. Surprising Results: Be cautious with results that seem too good to be true, as they often indicate errors or misinterpretations (Twyman's Law). Topics Covered: A/B testing, experimentation culture, failure rates, overall evaluation criterion (OEC), trust in experimentation, surprising results, Airbnb's experimentation approach, building experimentation platforms, P value misconceptions, starting with experiments, and institutional learning.