Lighthouse Related Product, an Efficient Cross-boundary Recommendation Engine on Qubole (Fanatics)

Speaker: Jing Pan, User Experience Researcher, Fanatics

Presentation: Fanatics, Inc. will introduce an item-to-item recommendation service platform in production, Lighthouse Related Product (LRP). LRP offers supervised-versus-non-supervised boundary-crossing and is extendable, flexible, and lightweight. LRP implements a modeling architecture that fuses into one system heterogeneous features from modern machine learning techniques of

1. non-supervised user-item matrices,
2. self-supervised Word2Vec, and
3. supervised XGBoost or deep learning.

This architecture allows innate extendibility to user-item recommendations, and flexibility for both offline and online use cases. It is lightweight and efficient enough to handle near one million products’ item-to-item recommendations on over 400 affiliated sites. The platform relies on the Apache Spark cluster in Qubole for both data feature extraction and prediction in a distributed manner with map procedure from a pre-trained supervised model; tasks on the Spark cluster in Qubole are seamlessly integrated into the rest of the workflows in Fanatics with another third-party scheduling service, Stone Branch. LRP has successfully passed the real-life load test of the 2017 holiday season and Super Bowl LII, and an earlier predecessor of the current version of LRP had achieved better performance in all measures, such as click-through rate and average order volume, compared to an industrial standard third-party recommendation service provider. Learn more at… Data Platforms Conference: https://www.dataplatforms.com Fanatics: https://www.fanatics.com Qubole: https://www.qubole.com