What is Big Data Analytics?

Big Data Analytics offers a nearly endless source of business and informational insight, that can lead to operational improvement and new opportunities for companies to provide unrealized revenue across almost every industry. From use cases like customer personalization, to risk mitigation, to fraud detection, to internal operations analysis, and all the other new use cases arising near-daily, the Value hidden in company data has companies looking to create a cutting-edge analytics operation.

Discovering value within raw data poses many challenges for IT teams. Every company has different needs and different data assets. Business initiatives change quickly in an ever-accelerating marketplace, and keeping up with new directives can require agility and scalability. On top of that, a successful Big Data Analytics operation requires enormous computing resources, technological infrastructure, and highly skilled personnel.

All of these challenges can cause many operations to fail before they deliver value. In the past, a lack of computing power and access to automation made a true production-scale analytics operation beyond the reach of most companies: Big Data was too expensive, with too much hassle, and no clear ROI. With the rise of cloud computing and new technologies in compute resource management, Big Data Analytics software is more accessible than ever before.

Big Data emerged from the early-2000s data boom, driven forward by many of the early internet and technology companies. Software and hardware capabilities could, for the first time in history, keep up with the massive amounts of unstructured information produced by consumers. New technologies like search engines, mobile devices, and industrial machines provided as much data as companies could handle—and the scale continues to grow.

In a study conducted by IDC, the Market Intelligence firm estimated that the global production of data would would grow 10x between 2015 and 2020.

With the astronomical growth in collectible data, it soon became evident that traditional data technologies such as data warehouses and relational databases were not well-suited to operate with the influx of unstructured data. The early Big Data innovation projects were open-sourced under the Apache Software Foundation, with most significant contributions coming from the likes of Google, Yahoo, Facebook, IBM, academia, and others. Some of the most widely used engines are:

• Apache Hive/Hadoop (developed at Yahoo!, Google, and Facebook) is the workhorse for complex ETL and data preparation that services information to many analytics environments or data stores for further analysis.
• Apache Spark (developed at University of California, Berkeley) tends to be used with heavy compute jobs that are typically batch ETL and ML workloads, but is also used in conjunction with technologies such as Apache Kafka.
• Presto (developed by Facebook) is a SQL engine that is lighting fast and reliable for reporting and ad-hoc analytics.

Typical Deployment

Business Intelligence is often times referred to as the first two descriptive and diagnostic stages of 4 steps to big data. BI is often hosted in a data warehouse where data is very structured in nature and only explains “what, where, and how” something happened (for example: 10 of the same shoes were purchased from 3 different stores that ran the same promotion, while the other 2 stores sold no shoes). This data is often used in reporting and gathering insights into popularity trends and interactions based on recent events.

Big Data Analytics takes this a step further, as the technology can access a variety of both structured and unstructured datasets (such as user behaviour or images). Big data can bring this data together with the historical information to determine what the probability of an event were to happen based on past experiences.

The 4 V’s have been a well known catalyst for the growth of Big Data analytics in last decade. Moreover, we have entered into a new era where new challenges are evolving like “variety” of open source technologies, Machine Learning use cases, and the rapid development across the big data ecosystem. These have added new challenges around how to keep up with the ever-growing information, while balancing how to ensure the effectiveness of advanced analytics in such a noisy environment.

Predictive and Prescriptive analytics is in a transient state, and requires modern infrastructure that traditional data warehouses can’t service. Having a big data platform that enables teams appropriate self-service access to unstructured data, enables companies to have more innovative data operations.

1. Descriptive analytics (What Happened and When) – This is common in traditional Business Intelligence and reporting analytics.
2. Diagnostic analytics (Where and How it Happened) – This takes Business Intelligence a step further, where the end user could be given a report or have a set of actions sent to them based on the results of the data.
3. Predictive analytics (What Will Happen and How) – Where a model is applied to the data and a decision or probability score is given based on historical events. This data can also be fed back into Business Intelligence systems to help with future decision making.
4. Prescriptive analytics (What Should We Do) – Takes the predicted output of the data and places it into a practical application that makes recommendations or alerts end-users (such as with fraud detection or ecommerce shopping). This data usually needs to be put into a data mart that can feed out to an application in near-real time.

Qubole is useful at any scale because the technology operates on the notion of separating storage from compute, and furthermore having managed autoscaling for Apache Hadoop, Apache Spark, Presto, and TensorFlow. The software automates cloud infrastructure provisioning, which saves a ton of time from data teams getting bogged down in administrative tasks such as cluster configuration and workload monitoring.

• ETL – build and schedule pipelines for recurring data transformations
• Data Science and Machine Learning – explore, develop and test models at scale before putting them into production
• Interactive Analytics – enable data teams and less technical users to analyze less structured or raw data that otherwise can’t fit in a data warehouse
• Data Visualization – format and present analytics for business insight and intuitive dashboards