Agenda

David Potes manages a team of Partner Solutions Architects at Amazon Web Services. He provides technical guidance, design advice and thought leadership to some of the most successful AWS software partners on the planet. His deepest expertise spans big data, security, storage and SaaS business applications. Over the last 15 years, he’s brought an intense customer focus to challenging and deeply technical roles in multiple industries. He holds both a world record and a number of patents.

The convergence of machine learning and the data lake offers businesses new opportunities to increase the pace of innovation, make better decisions and win new customers. AWS has an incredible array of services for analytics and machine learning, plus a broad and deep partner community of tools and services that give builders many options to choose from. In this technical session, we will look at patterns, practices and architectures that the world’s most innovative companies are using to change the future by predicting it today.

Building and running sustained global consulting organizations is Sarvottam’s forte. Having started his career with the industry and with deep consulting experience in a wide range of industries, his ability to bring process and IT together in a customer’s context is unique and much sought after in the high end CXO & disruptive consulting space. He has led various successful digital transformations with high impact for client organizations. His works on innovative data, analytics and AI product development, multi-country statutory requirements, process and IT integration models is widely acknowledged in the industry as path-breaking. Sarvottam is an avid reader, author and a life coach. He is currently an honorary advisory board member with Northwestern University , Chicago and likes to spend time in coaching and mentoring students spread across geographies.

Increasingly, businesses are realizing that data is not gold or oil, it’s lifeblood that they need to survive and thrive in the era of compressive disruption. Data and domain driven AI & analytics are the nerve centers that will keep businesses sharp and responsive.  Digital transformations are imperative in making organizations data-driven instinctive enterprises that are acutely aware & immediately responsive to their customers. This enables enterprises to become smarter, faster and move closer to their customers, outperforming competitors and capitalizing on opportunities. Data-driven instinctive enterprises emerge from a Connected ecosystem, Augmented Intelligence, and an Adaptive workforce. 

Mohan Naidu​ is Associate Director, Software Engineering at the Walt Disney Company, managing engineering for the Core Platform team for the last four years. Mohan has a successful track record of software engineering projects across various types of applications including Consumer Data Applications, (CDP), Master Data Management, cross-channel marketing applications, Spend Management solutions, among others at organizations like ESPN, Allied Informatics, Oracle, Toshiba, and SAP Ariba. In his free time, Mohan loves spending time with his family, watching movies, and enjoying cricket.

Today, enterprises have a desire to operate like a digitally native business where foresight powers intelligence, continuously informs decisions and monetizes data assets. This has always been the aspiration. Some enterprises have made great strides in this journey. For some, it has always remained a BHAG (bold, hairy and audacious goal). Some have fallen back on this arduous journey and at times losing out their differentiation. However, with recent advancements in technology, many enterprises can now re-align towards aiming for this goal. Join us to hear how modern enterprises would need an ambidextrous approach to building modern enterprises using Data and AI to drive differentiation in products and services.

Raman is a Vice President in Cognizant’s Artificial Intelligence and Analytics (AIA) practice. He heads the AI Centre of Excellence and Strategic Solutioning team. He also leads AIA Guilds & Communities in India, which is a key arm of the practice focusing on new age technology transformation in both Data and Intelligence (Cloud, Big Data, AI/ML, etc.)
With over 25 years of experience in helping customers in their Information Management and Analytics needs spanning geographies and industries, he also drives several strategic imperatives for the practice, helps with thought leadership, advisory and strategic solutioning for client’s digital data foundry, AI/ML adoption, AI solutioning, business case preparations, use case identification for potential AI applications and architecting enterprise grade digital data solutions.
He is a member of the advisory board of Analytics Society of India, a non-profit organization focused on promoting the usage and adoption of AI & Analytics across India.
Additionally, Raman helps shape practice strategies, supports in analyst relations, a regular speaker in conferences and seminars, a guest lecture in India’s No:1 B-school and periodically publishes papers and case studies.
His publication of a case study in Harvard Business Review is now one of the leading papers downloaded for teaching across 40 countries.

Dr. Kirk Borne is the Principal Data Scientist, Data Science Fellow, and an Executive Advisor at global technology and consulting firm Booz Allen Hamilton since 2015. He provides thought leadership, mentoring, training, and consulting activities in data science, machine learning, and AI across multiple disciplines. Previously, he was Professor of Astrophysics and Computational Science at George Mason University for 12 years in the graduate and undergraduate data science programs. Prior to that, he spent nearly 20 years supporting data systems activities for NASA space science programs, including a role as NASA’s Data Archive Project Scientist for the Hubble Space Telescope. Dr. Borne has degrees in physics (B.S., LSU) and astronomy (Ph.D., Caltech). He is an elected Fellow of the International Astrostatistics Association for his contributions to big data research in astronomy. In 2020, he was elected a Fellow of the American Astronomical Society for lifelong contributions to the field of astronomy. As a global speaker, he has given hundreds of invited talks worldwide, including keynote presentations at dozens of data science, AI and analytics conferences. He is an active contributor on social media, where he promotes data literacy for all and has been named consistently among the top worldwide social influencers in big data, data science, and AI since 2013.

Jorge A. Lopez is the Global Segment Lead for Analytics partners. Jorge has over 20 years of experience in analytics, business intelligence, and data integration. He holds an Engineering degree from ITESM in Mexico City, and an MBA from Virginia Tech. In addition to all things data, he enjoys listening to music (vinyl is best), reading, and exploring the outdoors.

Harsh is a seasoned analytics professional who has led BI & Analytics functions at organisations like Amazon, Swiggy and Uber before joining Angel Broking as Head of Analytics in Dec’19.
At Angel, Harsh and his team are focusing on building an Analytics as a Product (AaaP) platform that caters to real-time / near real-time applications that influence customer experience and behavior. The streaming data applications developed have grown significantly over the last 6 months and the team is now focused on building their push notification engine on top of the Qubole open data lake.

Today’s modern distributed data platforms are expanding at a quadratic rate in both size and complexity. It is more important than ever to define clear lines of delineation between compute and storage which allows both to scale independently. Scalability is becoming more and more important – both handing volume for a single data source as well as adding collections of disparate data sources into the data catalogue. Ultimately it is inevitable that data sources will be shared outside the platform to inside the platform and vice versa. In this presentation we will discuss the challenges with scaling storage and delve into the approach we have taken in The Walt Disney Company’s DTCI Data Platform to solve these complex issues in an easily maintainable, governable, audit-able, and simplistic way.

Mark Senerth has been solving problems for The Walt Disney Company for 15 years. He is well versed in cloud architecture, big data, distributed computing, and high throughput / low latency systems. Mark has the unique ability to see things from both a Macro and Micro perspective simultaneously; from the highest level architecture down the lowest level minute implementation detail. He has worked on a set of diverse datasets over the years that have been stored in an array of storage mediums – each having their own corresponding problem space. Currently Mark is focused on the Architecture and Implementation of the DTCI (Direct to Consumer & International) Unified Data Platform. The platform will enable the business to move faster and achieve better insights in a secure, governed ecosystem.

An enterprise data analytics and reporting platform typically runs data pipelines with long and complex jobs, spanning across many services, programs, tools, and scripts interacting together. These jobs need to run on an ad-hoc basis, have a set of dependencies on other existing datasets, and have other jobs that depend on them. Quickly this becomes a tangled mesh of computing and memory intensive processes, leading to a maintenance nightmare, instability, and poor performance. This calls for a need to build a scalable and optimized workflow management solution. While there are a plethora of open source solutions available to solve these problems, they may not fit everyone’s needs. So this talk provides an under-the-hood view into the architectural patterns of such solutions, and considerations for those companies that chose to build a more customizable, simple, and elegant solution without having to reinvent the wheel.

Shubham Tagra is a Technical Director at Qubole, working on Presto, Hive and RubiX development and making these solutions cloud optimised. Previously, Shubham worked on the storage area networks at NetApp. Shubham holds a bachelor’s degree in computer engineering from the National Institute of Technology, Karnataka, India.

As Spotad is supporting millions of queries per second, in order to make data reliable and easily accessible, a well designed data lake is one of our most important business aspects. In this presentation, I will be focusing on key aspects of data lakes architecture, cost and data based optimizations and clusters. It is well known that a well-partitioned data helps in reducing query costs and improving performance by limiting the amount of data a query needs to scan, in order to return the results. In particular, I’ll cover known and less known aspects of data partitioning, idempotency of data workflows and caching aspects to support your business goal.

Planning and optimizing are some of the strongest tools for maintaining a well-designed data lake, while keeping the cost at minimum and performance at its best. The most important aspect of those is to always know what is going on with your data. This includes monitoring query runtimes at all times, checking for the most and least queried data sources, checking clusters utilization and optimizing based on these results. I will discuss and demonstrate the importance of developing auto-monitoring tools and using the results for optimization. In addition to this, I will also discuss spot nodes utilization tools such as heterogeneous cluster nodes, and setting the maximum price in the context of cost-reduction and stability.

Rajat Monga is co-founder and former lead of TensorFlow at Google, powering machine learning research and products worldwide. As a founding member of the Google Brain team he was involved in designing and implementing DistBelief and more recently TensorFlow. Prior to this role, he led teams in AdWords, built out the engineering teams and designed web scale crawling and content-matching infrastructure at Attributor, and co-implemented and scaled eBay’s search engine. Rajat received a B.Tech. in Electrical Engineering from Indian Institute of Technology, Delhi.
He left Google recently to bring machine learning into some key aspects of the business world.

Enterprises migrating to the cloud for increased agility and elasticity is the new reality. As such, the migration of analytic workloads from on-premises data lakes to the cloud is part and parcel. With increasing compliance requirements and complexity of data platforms, the burden of implementing governance policies to comply with privacy mandates and industry specific regulations, like CCPA, GDPR, and HIPAA, largely falls on the shoulders of the data infrastructure teams. These teams are tasked with making data widely available to data scientists and business analysts, while simultaneously complying with external privacy regulations and internal best practices. To achieve this, comprehensive data access control policies need to be implemented to ensure only authorized users have access to data.

Dr. Srikanth Venkat is VP of Product Management at Privacera. Prior to Privacera, he built out Cloudera’s Shared Data Experience (SDX) portfolio which includes Apache Knox, Apache Ranger, Apache Atlas, and Hortonworks DataPlane Service. His prior career spans a variety of roles at Telefonica, Salesforce.com, Cisco-Webex, Proofpoint, Trilogy Software, and Hewlett-Packard. Srikanth holds a PhD in Engineering with a focus on Artificial Intelligence and an MBA. He enjoys tinkering with data science, cloud, and AI/ML technologies.

In 2019, ThoughtWorks published an article titled “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” which brings the concepts of domain-driven architecture to the data platform landscape. This presentation will cover the concepts and theory of what a domain-driven data platform looks like. We will then illustrate how we have architected and built a domain-driven platform in the cloud at The Walt Disney Company with lessons learned. This platform allows us to scale in ways previously impossible – particularly in thorny architectural areas such as migration, reorganization, distributed teams, and as the scope, size, and complexity of the data increases.

Caleb Jones has been at The Walt Disney Company (TWDC) for over 9 years working with large-scale data both as an engineer and architect. He is currently a Lead Architect for the Data Platform team in TWDC’s Direct to Consumer & International segment (DTCI) where he leads efforts to combine and unify data platforms which have powered several consumer brands. He has focused on creating a modern architecture that avoids the pitfalls of monolithic platforms, strongly unifies data, and provides autonomy to diverse teams. Besides data architecture, Caleb has a passion for analyzing connected data using graphs and networks and has helped to build some of the foundations to ID Graphs in TWDC.

Sean Knapp is the founder and CEO of Ascend.io. Prior to Ascend.io, Sean was a co-founder, CTO, and Chief Product Officer at Ooyala. At Ooyala Sean played key roles in raising $120M, scaling the company to 500 employees, Ooyala’s $410M acquisition, as well as Ooyala’s subsequent acquisitions of Videoplaza and Nativ. Before founding Ooyala, Sean worked at Google where he was the technical lead for Google’s legendary Web Search Frontend team, helping that team increase Google revenues by over $1B. Sean has both B.S. and M.S. degrees in Computer Science from Stanford University.

Hugo is the founding Partner of BigData4All, with more than 20 years of experience in business consulting and advising, where the last 17 he has specialized in supporting companies in the use and application of statistical and analytical models to improve business performance and uncovering actionable information.
He was Partner at Deloitte Mexico being responsible of Big Data & Analytics practice and was responsible for different practices in Consulting firms, such as KPMG and Capgemini, as well as leading different business lines in Teradata, IBM and Oracle.
He has been responsible for the design, development, implementation and operation of the information exploitation strategy for various companies in the consumer products, telecommunications, retail and finance sectors.
Hugo has a degree in Business Administration from the Universidad de las Americas, Puebla, where he also studied a Master’s Degree in Business Administration with a specialization in International Management. He has participated in executive programs for senior management, strategy and innovation.

Mr. Sanjeev Pant is the Vice President for Cloud Adoption, Strategy and Transformation Group with Coda, a Presidio Company. In this role Sanjeev is responsible to help and support customers larger initiatives to provide roadmap and guidance in transforming their digital footprint through a Cloud Journey. Sanjeev works directly with C-level executives to provide thought leadership and trusted insight into developing innovative projects.

In his past life, Sanjeev was the Practice Principal for Data & Analytics practice at Coda. In this role Sanjeev was responsible for successful engagement and delivery of customer-focused Data & Analytics projects. Before coming to Coda Sanjeev was the Chief Technologist for Data & Analytics at Cloudreach where he led various successful projects around Data & Analytics and Advanced Analytics (Machine Learning & AI).

Sanjeev has over 20 years of experience in various aspects of Information Technology and has successfully led and implemented huge enterprise applications that are both distributed and complex in nature.

These days Sanjeev spends a lot of time defining and implementing new Data & Analytics go-to-market strategies, and defining various successful Data & Analytics patterns that help his customers reach swift ROI and drive business outcomes.

Chris is the World Wide Head of Business Development for AWS Data Exchange, a new service that makes it easy for AWS customers to find, subscribe to, and use third-party data in the cloud.
Chris has deep experience in data licensing, third party data, and how data is used to in analytics and machine learning workloads to solve business challenges. Prior to joining AWS, Chris was the Global Head of Bloomberg’s Reference and Regulatory Data business, and spent nearly a decade in KPMG’s consulting practice where he assisted numerous Financial Services organizations operating in Australian, North American, and European markets with the implementation of transformation and regulatory compliance programs.

Zeotap saw a 10x growth with the number of data pipelines and amount of data processed growing at rapid pace in a short span of time. The increasing scale challenged our capability to track production data pipelines across multiple products and infrastructure deployments. To tackle this problem, we came up with Kingpin – a Centralized Job Management Framework. Kingpin addresses some of the common issues faced with data pipeline operations:

Additionally, we have integrated Kingpin with Data Visualization Tools for a Unified Job Level View with Metric Reporting capabilities to increase system visibility and reduce the operational time, efforts and costs involved.

In this talk, Sumit will talk about the current state of Apache Airflow and a couple of super cool features/enhancements of Airflow which were added very recently or going to be added into the near future, like Functional DAGs, Scheduler HA, Rest APIs, and more.

Get answers to your basic queries: What is Data Lake and why do people need it? What has changed in the recent years that people have started to look beyond data warehouses and data marts. Discuss a data lake architecture diagram – flow diagram and deep dive in Data lake characteristics (metadata, data lineage, cataloging, discovery, security). Understand next steps of your Data Lake journey i.e Visualisation , Real Time Analytics and Predictive Analytics.The session will end with Searce’s experiences and learning.

Ever had your CEO look at a report and say the numbers look way off? Has a customer ever called out incorrect data in one of your product dashboards? If this sounds familiar, data reliability should be the cornerstone of your data engineering strategy. In this talk, I’ll introduce the concept of “data downtime” — periods of time when data is partial, erroneous, missing or otherwise inaccurate — and how to eliminate it in your data lake, as well as the rest of your data ecosystem. Data downtime is highly costly for organizations, yet is often addressed ad hoc. We’ll discuss why data downtime matters to building a better data lake and tactics best-in-class organizations use to address it — including org structure, culture, and technology.

Businesses make many decisions to deliver values to their customers or users. As the key component of making decisions, not only should the business collect the right data, but they also should be able to use the data whenever decisions would be made. Therefore, it becomes crucial for businesses to have a system that allows them to do so, and as one of the e-commerce unicorns in Indonesia, Bukalapak understands that completely through a decentralized data platform. As an organisation that provides a product which has become a super app with millions of users every day, we need to be able to move fast with data to make the right decisions in the right time.

Migration from on-premise to the cloud was a moment that we used to make this transformation. With the growth of data in around tens of terabytes a day and thousands of queries executed per day, it in itself is a big challenge for us. We will show you the learnings from managing it on-premise that became the main reason as to why a decentralised model was a solution for us. Not only from the conceptual standpoint, changes happened on the other part of the architecture to make sure that we could provide the right service to our internal users.

On the ingestion part, we decided to replace Debezium and with a home grown CDC (Change Data Capture) and used Apache Beam with Google Cloud Dataflow in order to serve the data lake to our internal users. Then, in the downstream, we decided to utilise cloud services more by using Qubole for the batch processing. You might be able to see the reason behind the decision and the impact it brought to us for efficiency. On the presentation layer, we will show you how we implemented the lambda architecture to provide both delayed and fresh data to our internal users. Last but not least, to provide a better understanding of users’s experience, we created a quite comprehensive monitoring system to capture any problem earlier or even to understand any of users’s complaints or reports.

Ultimately, the success of fundamental changes on the system would be measured against what positive impact it brings to the organisation, therefore we would explain some of the benefits that it actually brought to BukaLapak, especially on how it increased our productivity by allowing us to get the data we need whenever decisions are meant to be made.

“Data lakes have been around for almost a decade since the term ‘data lake’ was first coined. The major breakthrough though was the advent of cloud around big data technologies which made data lakes more affordable and plausible. Data lakes have since evolved immensely over the years.
In the past they were huge reservoirs of data that were slow and batch oriented but now they are more like data hubs which are used to power transactional systems as well. They are more focused on speed and agility.
This talk would focus on the evolution of data lakes and how are they transforming the transactional and analytical world”

How do you drive business outcomes with Data? During this discussion, Coda, a Presidio Cloud Company, discusses several of our large North American customer engagements, how we approached these engagements, what were the concerns of c-level executives had from a business outcome perspective, and how did Coda leverage data designed principles to drive business outcomes. Additionally, during this session, Coda will be joined by a c-level executive customer to review how a data driven initiative transformed the business focus and enabled them to plan for future growth.
Potential customers: Wheel Pros; Greenway Health

Debugging slow spark applications when done with trial and error, takes lots of time. Sparklens provides insights about scalability limits of spark applications from a single run of the application. In this talk we will cover what Sparklens does and theory behind Sparklens. We will talk about how the structure of a spark application puts important constraints on its scalability. How can we find these structural constraints and how to use these constraints as a guide in solving performance and scalability problems of spark applications. Sparklens can answer if adding resources will decrease application runtime and by how much. It can answer if decreasing resources will increase efficiency and by how much. Sparklens makes the ROI of additional executor extremely obvious for a given application and needs just a single run of the application to determine how the application will behave with different executor counts. Specifically, it will help managers take the correct side of the tradeoff between spending developer time optimising applications vs spending money on compute bills.

GPS problems exist both at the edge and in the cloud. With current devices we can easily capture readings of latitude, longitude and elevation at every second of travel. These data can also be dressed by heart rate monitors and other sensors wired into our watches, wearables or phones. Putting these data in geographic context is an important part of providing value to the users. And it’s technically demanding. Maps provided by the USGS or other GIS services contain far more data than are required for a typical problem, and are usually packaged in a rather awkward, pixelated form. These discordant data structures present a host of problems for the data scientist. In this talk, I’ll walk through a typical GPS problem and the data science workflow around it. I’ll describe how big GPS data can be and how much noise it can contain. I’ll explore some useful products you might build with it, a practical guide to data management and introduce interplay between edge and cloud computing.

Advancements in the data and analytics space are commonplace nowadays, with the latest features and products getting the bulk of the attention. But how do we actually evaluate and utilize new features in the data space? This session discusses the importance of metadata, product tracking, and platform usage from a product manager’s point of view. You will also learn more about what drives product change by extracting platform insights to uncover users’ behavioral patterns. Lastly, we will take a look at changes in cost and education based on platform usage.

Presto revolutionized the data lake. This smart query engine enables organizations to quickly adopt an effective data lake architecture that supports a wide range of workloads and use cases. But everyone needs their “Prince Charming” and this is where indexing enters this fairy tale. Indexing has been widely adopted for database management, but it is only now starting to enter the world of truly big data.
In this session we’ll discuss:
● How you can effectively index at petabyte scale
● What is adaptive indexing
● What is dynamic indexing
● A reality check: which performance uplift you should expect

The growing volume of data requires skills to deal with dozens of new challenges like, how to ingest streaming mutable data? How to build and cache index for fast query? How to analyze data with ML? To address above challenges, we present CarbonData – a data storage that offers SQL API to ingest, query, and analyze data. It empowers ingestion with cloud-native ACID transactions and Streaming Merge SQL. Empowers Query with index, materialized view technologies, and a novel distributed index caching and pruning system that improves the query performance and outperforms existing cloud platforms. Empowers analytics with integrating data with ML frameworks, offers SQL API to track lineage and dependencies among data versions and models in the ML pipelines. The contributions are to call out the requirements of a high-performance data storage, to share experiences in exploiting novel technologies, to share our design in integrating data with ML by SQL, to discuss future challenges.

As more analytics workloads shift to the cloud, more customers are motivated to “unstick” their on-premises data warehouse, data lake and analytics platforms. Join us to learn how Google Cloud and our Analytics partners play a critical role in helping enterprises through their digital transformation journey.

Acquia Marketing Cloud helps marketing teams understand, personalize and engage with users on any channel. Our product suite is open, data first and machine learning forward. For us, this translates into challenges like supporting a multi-cloud architecture, building data pipelines at scale, or creating Machine Learning driven experiences at every touchpoint . Join us to discover how Acquia Marketing Cloud is overcoming these challenges in the real world.

Data environments are complex but your data technology stack doesn’t have to be. Advances in data lake and analytics technologies now allow data teams to simplify their data stack, reducing unnecessary transformation, accelerating data pipelines, and refocusing efforts on delivering insights to the business. This session will outline the modern data stack, how data analytics tools can help simplify data access, and new ways data teams can put data into the hands of those who need it most.

The last decade has brought great advancements and innovations across the data management landscape, spanning large-scale processing engines, cloud-based data warehouses, next generation data lakes and unprecedented machine learning tools. But why are these powerful systems being controlled and orchestrated by the most rudimentary technologies? Data movement and transformation are often still dictated by manual, hard-coded triggers and rules, written in slow development cycles and resulting in brittle pipelines. As the number of pipelines and dependencies scale, data engineering teams become bogged down by constant maintenance, combing through code and logs to find hotspots, and constantly tuning infrastructure just to keep things running. This “state of the art” points to a gap in the modern data ecosystem, opening the door to more intelligent forms of data orchestration. As was done with Kubernetes, Terraform, and even React, we can apply a declarative approach to the domain of data pipelines, radically advancing the level of automation that can be achieved. In this session, Sean Knapp, CEO & founder of Ascend.io, discusses this journey, including:
* Imperative vs Declarative systems
* Evolution of orchestration tools
* Tradeoffs and design decisions in moving from task-centric to data-centric architectures
* Challenges in running a multi-cloud, massive-scale, elastic infrastructure built on Kubernetes and Apache Spark

Case study where a major worldwide bakery company decided to migrate from their 15-year-old legacy Datawarehouse to a Data Lake on Cloud migrating more than 100 workloads in a 1 year-span, obtaining critical benefits form cost reduction, faster time to market, lower TCO and increased final user satisfaction.

This talk is about MiQ’s journey to build a scalable Data and Analytics Ecosystem. It covers a saga of various decisions and learnings wrt to technology and product that MiQ has seen over a decade. It covers the evolution of data and related choices to enable what we call today a scalable Programmatic Media Buying space.

This talk focusses on the journey of building a data minimization algorithm and its challenges to build a data governance system for GDPR world. It showcases the minimization approach along with certain data pattern problems and how they are solved.

As technology evolves, we find ourselves surrounded by more and more data. No longer can we manage this data directly. We must rely on powerful analytic techniques (Artificial Intelligence, Data Science, Deep Learning, etc.) to derive value at a pace that matches that of which data is produced. This can be a difficult task, further burdened by the desire to have real-time results. If you find yourself worrying about how to minimize the time it takes your pipeline to compute results, take a deep look at your data model. In this talk, we explore various ways that Orbital Insight strives to push the boundaries around real-time geospatial analytics through intelligent pre-processing and partitioning, predictive ordering, trading off generality and functionality, and more.

Multitenancy in Datalakes are on the rise. When looking at multi tenancy from the lens of data governance a lot is changing the landscape and the way we have been operating with respect to the governance model probably needs a rethink. It is time to think of Governance and its various entities as a first-class citizen in data architecture and bake it as part of the platform. We will look at the various aspects of governance extending to accommodate the growing compliance and regulatory requirements and suggestive architectural approaches to realize the same.

As enterprises accelerate their timelines towards digital transformation, they’re looking for seamless ways to move their on-prem data and analytics workloads to the cloud. Open solutions like Qubole are integrated with Google Cloud tools like Looker and BigQuery–meaning they can help enterprises scale. Join us to discuss how your organization can best leverage Qubole on Google Cloud.

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at companies such as Comcast, FINRA, LinkedIn, Lyft, Netflix and Pinterest, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Storage, HDFS, NoSQL and RDBMS data stores.

We will present the recent Presto advancements and the project roadmap going forward, and then detail Presto’s support for Apache Iceberg, an open table format for huge analytic data sets, optimized for modern data lake storage systems.

This talk is about MiQ’s journey to maintain a sustainable, efficient & cost effective Data Lake solution. It covers cost optimization initiatives taken in space of data pipeline as well as infrastructure. A deep dive on the recent approaches to make cost reporting and debugging automated and self served.

In most modern e-commerce, a microservice architecture drives continuous business while large scale Big Data applications add value to them from time to time. There is usually a clear boundary between these two worlds with a narrow stream of information moving across the barrier. With Data Lake architecture and the ecosystem around it maturing, there is no reason for this barrier to exist anymore. We must strive to move towards a world where we build applications that are built over data lake and a continuous stream of data that can provide real-time decisioning power to the rest of the stack. In this talk, we discuss about ideas and practices that allow us to build Data Products that are powered by Big Data but are lean and lightweight to be integrated directly to Microservice based architectures.

Cloud stores are inexpensive and infinitely scalable which has lead to them being the de facto standard for data lakes. But when it comes to performance, local SSDs easily outperform them especially with newer advancements like NVMe. Furthermore, due to the access over network cloud stores often struggle to provide consistent performance and in certain cases, like inter-region network access in AWS, can turn out to be expensive. Given that all cloud providers have an option to provision machines with high speed local disks, many of the shortcomings of the cloud stores can be avoided by using these disks for a cache. RubiX is Qubole’s home grown, open source data caching framework that integrates with Big data engines like Hive, Presto, Spark to provide the data cache over any cloud store.
In this talk we will find out how RubiX works, how it integrates with the Big Data engines and the cloud store, what kind of improvements can be expected and what new features are being worked upon.

Organizations that use data as a competitive differentiator are more likely to lead and outperform their peers. Many companies have transformed their data architectures and adopted the cloud to meet a variety of scalability and operational challenges. In this session, we will talk about some of the most relevant data analytics trends and architectural patterns – from data lakes to machine learning – that are helping organizations innovate faster and stay ahead of the competition.

Qubole and Nextdoor have been partners for a while: we’ve leveraged Airflow and Qubole extensively to create ETL-pipelines-as-configuration, democratizing the data platform and scaling effectively with company growth. However, we quickly realized that our data warehouse — housed on AWS Redshift, and what powers our Tableau dashboards — could not meet the demands of rapidly-changing schemas and analytics query volume. Instead of investing in a data warehouse cluster capable of meeting the volume and throughput, we instead doubled down on the data lake architecture, moving to Looker to power our dashboards, and Qubole as the data warehouse engine. In this talk, I’m going to argue that the concession of processing performance in decoupling storage and compute pales in comparison to the data discoverability and the analytics velocity gained from an open data lake, as well as our journey to get here.

Capillary’s Multi-Tenant Data Lake powers 250+ enterprise retailers and consumer brands in a SaaS model on the cloud. The data lake, built on Apache Spark, provides the “Consumer Data Platform” for our customers of various verticals like Retail, Grocery, FnB and Jewelry, etc. This data platform in turn generates the essential insights which forms the base for effective marketing engagement and eventually leads to consumer retention and loyalty. Any multi-tenant platform must cater to commonality and uniqueness of its tenants. Similarly, this platform deals with each customer’s unique analytics requirements, data schema and data availability. The talk focusses on the key aspects of modeling such a system around the modern data lake concepts.

In a covid-19 world, companies are looking for ways to reduce cloud spending as much as possible. While many Apache Spark tuning guides discuss how to get the best performance using Spark, none of them ever discuss the cost of that performance. In this session, we’ll cover a proven tuning technique for Apache Spark that lowers job costs on AWS while maximizing performance.
Topics include:
* the principle for how to make Apache Spark jobs cost efficient
* how to determine the AWS costs for your Apache Spark job
* how to determine the most cost efficient executor configuration for your cluster
* how to migrate your existing jobs to the cost efficient executor
* how to improve performance with your cost efficient executor

There have been amazing advances in AI over the last decade. These have led to huge impact, but what’s driven these advances? These questions will help us answer why the impact seems to be much more consumer centric, and what it would take to bring it to businesses. Does the same technology that delivered Google Photos and Translate, have something for the enterprise as well?

Adobe Digital Experience group has created the Experience Platform, the foundation of Experience Cloud products, as an open system that transforms all your data —Adobe and non-Adobe — into robust customer profiles. Through the development of Experience Platform, data pipeline engineering has emerged as a key organizational capability in line with developments across the delivery layers for software services. Consider: system layer transitions to Cloud, Containers, k8 environments, platform layer with the rise of Distributed Service architectures and API Ecosystem application layer where the expectation is a predictable and accurate Machine Learning system

A consequence of these changes the need to alert and monitor the quality of data processing and movement across cloud and on-premise environments with compute and storage services working in both streaming and batch modes. The expectation is that complex data processing workloads are handled by pipelines which are fault tolerant, repeatable, and highly available. Customers demand that we ensure resource availability, manage inter-task dependencies, retry transient failures or timeouts in individual tasks, or create a failure transmitted via notification system. The expectation that solutions involved in complex pipelines will support interfaces allowing integration of the solution into the broader pipeline. This article discusses how to use AEP’s Webhook interfaces to automate dataset monitoring.

Before co-founding Qubole, Ashish ran Facebook’s Data Infrastructure team; under his leadership the team built one of the largest data processing and analytics platforms in the world. This platform achieved not just the bold aim of making data accessible to analysts, engineers and data scientists, but drove the “big data” revolution. In the process of scaling Facebook’s Big Data infrastructure, he helped drive the creation of a host of tools, technologies and templates that are used industry wide today.

Eddie is the Partner Development Manager at Google Cloud where he is focused on developing innovative technology and go-to-market motions with some of Google Cloud’s key Data Analytics Platform partners. He is a seasoned executive with recent stints at Pentaho, Hitachi Vantara, and Dremio.

Tom has 20+ years experience working in SaaS, Mobile, Advertising, and Video startups. Introduced to Hive in 2009, the focus of the last 10 years has been large scale data systems and cloud/hybrid environments with an emphasis on data quality, data pipeline management, and operational excellence. Currently, he is helping Adobe Digital Experience customers onboard onto the Adobe Experience Platform. AEP is an innovative project which brings together the insights from Digital Advertising/DSP, Web site Analytics, DMP, and Email marketing to provide unparalleled user-based marketing insights and activation opportunities.

Engineering Lead with an expertise in working on Programmatic Media Buying space for Ad-Tech domain. Skilled in Application Development and building a highly scalable Big Data Analytics platform. Data Pipeline builder responsible for optimizing data stores and building team for the same from the ground up. Experienced in performing root cause analysis and optimizing Data Pipelines at scale with cost effective approach.

Prabhu is the Chief Technology officer at MiQ Digital, a leading programmatic partner across channels, screens in the ad-tech space. Prabhu has helped build from ground up MiQ’s tech and analytics ecosystem which delivers products and solutions driving its global business. As MiQ has grown, he is helping MiQ’s tech and analytics capabilities evolve whilst ensuring operational excellence in delivering world class products continues to scale. He has two decades of experience in the software industry and has designed and developed high performing large scale backend platforms and complex enterprise applications.

Shreya Pal is a Technologist with 17+ Years of experience in Big Data, Cloud and Analytics. She has rich experience in architecture of large-scale distributed systems, designing complex solutions and building services in cloud technologies. Shreya is a decisive leader in Big Data space with expertise in execution of large scale programs , Big Data strategy, consulting, CoE setup and pre-sales. She has worked across global businesses for Fortune 500 customers in Banking, Life Sciences, Media & Entertainment to define their Big Data roadmap.

Brad Caffey is a Staff Data Engineer at Expedia Group who has over 20 years of data engineering experience. He particularly enjoys doing deep analysis of complex data engineering problems. Outside of work he enjoys spending time with his wife and kids as well as rooting for his Texas Longhorns.

Matt leads Orbital Insight’s Engineering team. He has more than a decade of engineering and client facing experience. Matt joined Orbital Insight via its acquisition of FeatureX, where he was CTO and, previously, Chief Architect. Matt previously worked as a Quantitative Software Engineer at Two Sigma and a Forward Deployed Engineer at Palantir, following his time at The MIT Lincoln Laboratory and Facebook. Matt earned an MS in Computer Science from MIT and BS in Computer Science, Mathematics, and Physics from MIT.

Lead Engineers with demonstrated history of working in the ad-tech industry along with the challenges and scale it brings. Skilled in Application Development and building highly Scalable Big Data Analytics platforms.

Javier is the author of “Mastering Spark with R”, sparklyr, mlflow, torch and many other R packages for deep learning and data science. He holds a double degree in Math and Software Engineer and decades of industry experience with a focus on data analysis. He currently works in RStudio and previously in Microsoft Research and SAP.

Engineering Leads with an expertise in working on Programmatic Media Buying space for Ad-Tech domain. Data Pipeline builders responsible for optimising data stores and building team for the same from the ground up. Experienced in performing root cause analysis and optimising Data Pipelines at scale with cost effective approach.

Brad is passionate about making data science easier by empowering developers with better tools and improving societal understanding. He is currently a Developer Programs Engineer at Google where he specializes in machine learning and big data solutions. Outside of work, Brad can be found singing, riding his bike and playing Magic the Gathering.

Lead Engineers with demonstrated history of working in the ad-tech industry along with the challenges and scale it brings. Skilled in Application Development and building highly Scalable Big Data Analytics platforms.

I am a founding member of Swiggy’s Data Platform which has grown exponentially over the last 3 years. Since then, we have utilized Big Data to make real-time business decisions to deliver a great experience to our customers.

Before co-founding Qubole Joydeep worked at Facebook where he boot-strapped the data processing ecosystem based on Hadoop, started the Apache Hive project and led Facebook’s Data Infrastructure team. Joydeep was also a key contributor on the Facebook Messages architecture team and brought the power of Apache Hbase to Facebook and to the transactional and reporting backends for Facebook Credits.

Sirish is an Engineering leader at Orbital Insight with 18 + years’ experience in the field of software engineering and core focus on building robust and resilient distributed applications, APIs and cloud-based services. He is a big proponent of event based Microservices. Sirish is also passionate about running customer-obsessed engineering teams that can consistently deliver world-class quality software on time. His favorite quote is “Change is the only constant.”

RK’s favorite quote: “Talk is cheap, show me the code”. As the Chief Technology Officer, RK leads Searce’s Cloud strategy and solutions capabilities. With over 10+years of rich IT experience and previously co-founding companies, nothing gets him more excited than solving complex infrastructure problems for clients. RK specializes in delivering best-in-class solutions for big data and analytics, application migration, infrastructure automation thus reinforcing a strong engineering culture at Searce.

David Phillips is a co-creator of Presto and CTO at Starburst. Prior to Starburst, David was a Software Engineer at Facebook, and held senior engineering positions at Proofpoint, Ning and Adknowledge.”

Pravanjan has been responsible for taking Capillary’s product vision forward and ensuring partner brands always stay consumer-ready through the solutions the company provides. In his time at Capillary, he has acted as the chief architect of the company’s data-driven technology stack.
With a doctorate and bachelor’s from IIT, Kharagpur, Pravanjan was the principal architect at Silicon Valley startup, Minekey where he developed products around content personalization, recommendation, search, social collaboration, and microblogging. He has also worked with several research groups and acted as a consultant for National Semiconductors where he helped them in the domain of Embedded Systems design and verification.

Hafiz Badrie Lubis is the AVP of engineering for Data and Discovery in Bukalapak. A graduate from Informatics engineering of Bandung Institute of Technology. In his entire 12 years of experience, he had contributed, in the domain of data engineering, to multiple companies, including the big ones like Vidio, BBM (BlackBerry Messenger), and Bukalapak.

Ivan is a sofware engineer at Nextdoor on the Data Platform Team. His main responsibilities include developing and maintaining the data infrastructure used to process the various ETL pipelines at Nextdoor. He’s particularly passionate about functional programming and thinks everything can and should be written in Scala. When he’s not coding or not sheltering-in-place, he can be found playing ultimate Frisbee or cycling wherever he can with his single-speed bike.

Kent Buboltz is a Product Manager at Expedia Group, where he climbed the ranks as an analyst before moving to London to manage the analytics platform with Hotels.com. He currently works with various analytics platform vendors to provide the best experience to all types of platform users across Expedia Group. Kent holds a B.S. in Mathematics and Global Business from the University of Redlands and a M.S. in Project Management from Boston University. In his free time, Kent is a mobile app developer, avid traveler, and is always searching for the best cup of tea.

Martin Traverso is a co-creator of Presto and CTO at Starburst. Prior to Starburst, Martin worked as a Software Engineer at Facebook, and a Software Architect at Proofpoint and Ning.

Data/mathematics handyperson for those who might need it. I’m also bringing mathematical physics to the masses via YouTube. But I can’t make them care, so there’s also adventure content.

Working closely with a talented team of Product Analysts, Business Analysts, Data Engineers & Data Scientists. I am responsible for leading & coordinating the data generation, ingestion, transformation & insight generation across the organisation. Working with leadership in democratising data for unblocking data-driven decision making. Building platforms for data science & advanced analytics use cases. Have in-depth industry knowledge in FinTech, Retail, Technology domains.

Barr Moses is CEO & Co-Founder of Monte Carlo, a data reliability company backed by Accel and other top Silicon Valley investors. Previously, she was VP Customer Operations at customer success company Gainsight, where she helped scale the company 10x in revenue and among other functions, built the data/analytics team. Prior to that, she was a management consultant at Bain & Company and a research assistant at the Statistics Department at Stanford. She also served in the Israeli Air Force as a commander of an intelligence data analyst unit. Barr graduated from Stanford with a B.Sc. in Mathematical and Computational Science.

Ori Reshef brings 15+ years of experience and deep expertise in the data space, specializing in speech, text and big data analytics, and creating business solutions from cutting edge technologies. As a product expert, Ori helped shape various solutions that focus on improving customer lifetime value (LTV) by mining three primary types of data:

Prior to joining Varada, Ori was the VP Data Products and Head of Data Science for Clicktale (acquired by Contentsquare), led the Analytics & Intelligence product for LivePerson (NASDAQ: LPSN), as well as several other senior product positions. Ori transitioned to executive product roles after serving as a Solution Delivery Manager and Senior Sales Consultant for Nice Systems (NASDAQ:NICE) and several other technology consulting positions.

Joel McKelvey is the Head of Looker Core Product Marketing at Google. In this role he serves as a subject matter expert on both analytics products and the enterprise adoption of data-focused technologies. He has a comprehensive background in data and analytics, including storage technologies, database architectures, analytics deployment models, and industry trends. Mr. McKelvey is an experienced presenter, having spoken on technical and business topics internationally at numerous conferences and events.

Arnaud Prades is the Director of Machine Learning for Acquia Marketing Cloud. He specializes in leveraging data and software to build cool stuff that works in fields such as customer intelligence, logistics, dynamic pricing or algorithmic trading. He also likes cats and kitesurfing.

Ajantha Bhat is an Apache CarbonData committer & Senior Tech Lead at Huawei Technologies.

Rohit Karlupia has been mainly writing high performance server applications, ever since completing his Bachelors of Technology in Computer Science and Engineering from IIT Delhi in 2001. He has deep expertise in the domain of messaging, API gateways and mobile applications. His primary research interests are performance and scalability of cloud applications. At Qubole, he focused on making Big Data as a Service, debuggable, scalable and performant. His current work includes SparkLens (open source Spark profiler), SparkLens based auto scaling, Logan/DeLog: accelerated debugging tool for anomalus log message detection and safe feature rollouts.

David has been responsible for carrying out Spotad’s BI and data agenda, and for ensuring that data is always rapidly and comfortably accessible to all external and internal company’s interfaces. In his time at Spotad, he has acted as a member of the data team and worked in various roles, mainly related to BI.
With a bachelor’s degree from the Interdisciplinary Center (IDC), Israel, David participated in a collaborative project between one of the largest telecommunications companies in Israel and the IDC, in which he led the development and implementation process of a model for predicting employees’ turnover.
He has also worked on developing a predictive model for bid price forecasting, as part of the Data Science team in Spotad.

Sathish heads engineering at Zeotap. His responsibilities also include engineering strategy and technical architecture. He comes with 18+ years of experience and has been building big data stacks for various verticals for past 8 years.

Sumit is currently a Tech Lead in Twitter’s Data Platform group and an Apache Airflow PMC member as well. In his 12yrs of industry experience, he spent most of the time in building SAAS products around Big Data technologies. At Twitter, he leads the workflow team which is responsible to provide managed Airflow as a service to other teams across Twitter engineering.

Shantanu Shrivastava, a Senior Software Engineer at Zeotap, is part of the Data Engineering Team which handles massive scale of data on a daily basis. He works on zeoCore, the core data platform, responsible for ingesting, refining and distributing data from multiple partners across a wide range of platforms. He has a keen interest in building and optimizing data pipelines and working with distributed systems. Shantanu holds a bachelor’s degree in Computer Science from BITS Pilani