So, early in the product-development process, TrafficGuard turned to Qubole to provide a cloud-native data processing infrastructure that would guarantee the scalability and cost efficiencies required.
“Ad fraud is trying increasingly to resemble human behavior,” says Head of Data Science and Analytics, Raigon Jolly. So, “in order to reliably detect something that is continuously evolving, the technology must be very sophisticated. We have a rapid rate of development so we needed tools that simplify and streamline infrastructure management so that we can focus on developments that our clients directly benefit from.”
TrafficGuard is a software as a service (SaaS) product that detects, mitigates, and reports ad fraud in real time through the use of several sophisticated machine learning (ML) models. TrafficGuard is owned by Australian-based Adveritas Limited, which specializes in big data technologies and Ad Tech software solutions. Find more information about TrafficGuard at https://trafficguard.ai.
Detecting and preventing ad fraud in near-real time means processing considerable amounts of data. Rather than attempting to build and manage the architecture themselves, the TrafficGuard team opted to partner with Qubole from the start.
Senior engineer Pravin Todkar says that Qubole enabled TrafficGuard to bring their innovative fraud prevention to market much more quickly. “It would have been really difficult to build TrafficGuard without Qubole,” he observes. Moreover, Qubole empowered the TrafficGuard team to “focus on product innovation rather than infrastructure management,” says Todkar. This has yielded concrete benefits, like quicker development turnaround times. It has also resulted in less tangible rewards, such as fostering a culture of innovation – which in turn inspires team members to take risks and helps attract and retain top talent. Faster time to market and an innovative culture – that’s a recipe for success.
“Qubole has helped us federate data, manage data pipelines, streamline infrastructure management and leverage open source technologies to support our efforts in building enterprise-grade ad fraud prevention.” -Pravin Todkar, Senior Engineer, AI, Data, and Cloud Services, TrafficGuard
TrafficGuard processes approximately 1 billion data transactions a day – roughly 10 terabytes and is rapidly scaling, over the last 6 months they have seen a 12 X increase. Some of these data transactions are essentially constant and require always-on clusters. But other data transactions spike at unpredictable times. For these, TrafficGuard employs AWS Spot instances through Qubole’s Intelligent Spot Management capabilities. “Spot instances are really helpful,” says Todkar. “With spot instances, it takes only a second to spin up clusters and start running workloads.”
Spot instances improve efficiency and keep costs down. They achieve this by aggressively downscaling as soon as the workload is complete – meaning that TrafficGuard never pays for idle clusters. And because all this upscaling and downscaling occurs automatically in Qubole—based on workloads, job priority, or SLAs – the company saves on labor costs, too. That is, rather than taking on new hires to manage all these (and other) operations, “we’ve been able to achieve more with the DevOps resources we currently have,” says Jolly.
On top of all that, says Todkar, “Qubole has an excellent customer support team. Their expert opinion comes in handy at times when we face technical issues.”
“We are a growing business with new clients coming onboard frequently. The nature of digital advertising is that traffic volumes can be volatile, fluctuating with little warning. For fraud detection and other business needs, Qubole handles fluctuations of data with autoscaling. Spot instances are really helpful in terms of managing cost.” -Pravin Todkar, Senior Engineer, AI, Data, and Cloud Services, TrafficGuard
Many solutions designed to combat ad fraud rely on tools like rules engines and IP blacklists to detect it. But, these tools present two critical limitations. First, these tools look for known indicators of ad fraud, so are not suitable for detecting new fraud tactics as they evolve. This leaves advertisers exposed to new forms of fraud. Second, these tools may flag valid impressions, clicks, or install events as fraudulent. These false positives can result in valid traffic being removed, legitimate supply sources not receiving due payment, and advertising campaigns’ effectiveness compromised.
To overcome these limitations, the TrafficGuard team has turned to sophisticated machine learning models. Management of data pipelines and infrastructure to support this effort is streamlined with Qubole. These models analyze combinations of indicators over time and across devices to detect fraud as it evolves as well as mitigate false positives. When used in concert with rules engines and blacklists, TrafficGuard’s models provide far greater protection against both known and unknown forms of ad fraud.
So far, the TrafficGuard team has developed more than 10 machine learning models for use in fraud detection—a number “that is expected to increase significantly in the near future,” says Todkar.
“Given our fast pace of growth, our data needs are ever increasing. Scale and reliability are integral to our solution, as is continuous innovation to keep ahead of sophisticated fraud. Using Qubole, our infrastructure automatically adapts so the team can focus on developments that our customers can see.” -Pravin Todkar, Senior Engineer, AI, Data, and Cloud Services, TrafficGuard