Building a Real-Time Decision Engine Using ML on Apache Spark Structured Streaming

Speaker: Garren Staubli, Sr. Data Engineer at BluePrint Technologies

Abstract: Real-time decision making using ML/AI is the holy grail of customer-facing applications. It’s no longer a long-shot dream; it’s our new reality. The real-time decision engine leverages the latest features in Apache Spark 2.3, including stream-to-stream joins and Spark ML, to directly improve the customer experience. We will discuss the architecture at length, including data source features and technical intricacies, as well as model training and serving dynamics. Critically, real-time decision engines that directly affect customer experience require production-level SLAs and/or reliable fallbacks to avoid meltdowns, which this talk will also address. See more about the event at https://www.dataplatforms.com/