Senior Data Scientist/Statistician - Experimentation

Introduction

Rigorous statistical analysis is a core to all experimentation products at Optimizely. In this role, you will get on customer calls to articulate how we maintain statistical accuracy in our products, as well as triage questions around statistical inference. You will educate internal and external stakeholders about fundamental as well as cutting edge statistics concepts via webinars, speaking engagements, blog posts, and white papers.  You will also have the opportunity to work with our product and engineering teams to build new statistics/ML features for all of Optimizely’s products.  You will have also have the opportunity to partner with experts in academia and research to explore and identify the latest trends and next great innovations that Optimizely should invest in. 

Job Responsibilities

Product and Feature Development 

• Collaborate with product managers and engineering teams to design and implement new statistical and machine learning features across Optimizely’s product suite. 
• Conduct rigorous testing and validation of new statistical methods and ensure seamless integration into the platform. 
• Analyze product performance data to optimize existing features and provide actionable recommendations for improvement. 

Research and Innovation 

• Partner with academic and industry researchers to explore cutting-edge statistical methods and machine learning innovations. 
• Identify emerging trends in experimentation and statistics that could enhance Optimizely’s offerings. 
• Propose and prototype advanced statistical techniques to solve challenging business problems faced by customers. 

Quality Assurance and Compliance 

• Develop and enforce statistical standards and best practices for all experimentation processes. 
• Monitor and evaluate statistical algorithms to ensure accuracy, reliability, and scalability. 
• Provide internal stakeholders with guidance on statistical compliance and methodologies to safeguard the credibility of experiment results. 

Education and Thought Leadership 

• Develop and deliver educational content on statistics, including webinars, blog posts, white papers, and public speaking engagements. 
• Simplify complex statistical concepts for both technical and non-technical audiences to promote understanding of experimentation best practices. 
• Represent Optimizely at conferences and industry events, showcasing expertise in experimental design and statistical analysis. 

Customer Engagement and Support 

• Conduct customer calls to explain how statistical accuracy is maintained across Optimizely’s experimentation products. 
• Triage and resolve customer inquiries related to statistical inference and experimentation methodologies. 
• Assist customers in designing statistically valid experiments tailored to their needs and objectives. 

Cross-Functional Collaboration 

• Partner with sales, customer success and onboarding teams to offer statistical expertise during customer onboarding and key engagements. 
• Serve as a statistical advocate within the organization, fostering a culture of rigorous data-driven decision-making. 
• Collaborate with marketing teams to showcase Optimizely’s statistical capabilities in external communications. 

Knowledge and Experience

• 3+ years of industry or educating experience with experimentation, data science, analytics, etc. 
• Understanding of and ability to explain experimentation, randomized control trials, and inference, particularly aspects of high throughput testing such as multiple hypothesis testing, sequential testing, reinforcement learning, robustness, data mining of experiments 
• Demonstrable experience of at least one of the following subjects: 
  • Classical statistical inference (fixed horizon tests, sequential tests); 
  • Bayesian statistical inference; 
  • Causal inference (potential outcome framework, randomized controlled trials, observational studies) and experiment design; 
  • Reinforcement learning (e.g., multiarmed bandit). 
• Demonstrable working experience of python and SQL, (knowing R is also helpful). 
• Experience collaborating on applied statistical research or designing commercial statistical features/applications 

Education

Advanced degree in Statistics, Computer Science or a closely related field such as bioinformatics, econometrics, economics or applied mathematics