Year 1 of an experimentation platform team: challenges, achievements, and lessons learned

Published by Rommil on

We share our experiences on the Zulily Experimentation Platform squad, formally instituted 1 year ago as part of Zulily’s investment in scaling up the quantity, quality, and velocity of online experimentation across all customer touchpoints. We discuss how we managed to move away from a collection of ad-hoc, manually-curated, labor-intensive readouts and to standard templates, reliable engineering processes, and self-service workflows. We also instituted a program of cross-functional education of on experimentation, which started with a group reading of “the book” by Kohavi, Tang and Xu and is now supported by an experimentation clearinghouse process through a center-of-excellence. We share successes, challenges, temporary setbacks and achievements: during year 1 as a dedicated experimentation team, we increased the experimentation throughput by over 200%.