Many companies are well aware of the benefits big data can bring to their organizations. Gaining a better understanding of customer behavior and making better business decisions are just a few of the many benefits data-driven organizations are capable of achieving. Yet many companies still struggle when it comes to understanding how to effectively leverage big data.
While it’s clear that data-driven organizations perform better by making business decisions based on data-driven inisghts, most companies are still wondering “how” to get the most of out of their big data analytics. According to a recent Gartner survey, investments in big data have increased, but organizations still find it difficult to achieve the benefits. Most companies are actually failing when it comes to leveraging big data, as only 15 percent of business reported deploying their big data projects to production (unchanged from last year at 14 percent).
“Big data is a collection of different data management technologies and practices that support multiple analytics use cases. Organizations are moving from vague notions of data and analytics to specific business problems that data can address. Its success depends on a holistic strategy around business outcomes, skilled personnel, data and infrastructure,” said Nick Heudecker, Research Director at Gartner.
What About DataOps?
Figuring out how organizations can achieve a truly data-driven enterprise struck a nerve with Ashish Thusoo, Co-Founder and CEO of Qubole. Ashish is an entrepreneur who previously led Facebook’s Data Infrastructure team and helped create Apache Hive. Under Ashish’s leadership at Facebook, his team built one of the largest data processing and analytics platforms in the world.
During his time at Facebook, Ashish also pioneered innovative ways to adapt popular open source big data technologies to work optimally in the cloud. This inspired him to found Qubole, an autonomous data platform based in the cloud, which allows enterprises to leverage big data as a self-service.
“Understanding how to take full advantage of big data requires a mixture of making smart technology decisions and understanding the importance of running analytics in the cloud. The roles of people, processes and culture are also crucial elements for mastering the data-driven enterprise,” said Ashish.
Ashish recently co-authored a book on creating a data-driven enterprise with “DataOps,” which is the equivalent of DevOps for data. DataOps is a revolutionary way of managing data that promotes communication between formerly siloed data, teams and systems.
Just as DevOps focuses on people, processes and technology, DataOps takes advantage of process change, organizational realignment and technology to facilitate relationships between everyone who handles data - data scientists, engineers, developers, business users, etc. – allowing all users to have access to data on the fly.
DataOps And The Cloud
Many enterprises might still be wondering, “How does a DataOps approach leverage big data to achieve a data-driven organization?”
According to Ashish, the cloud is the best way for data-driven enterprises to efficiently run analytics workloads. If a company uses their own on-premises data center, it could take 16-18 months to get to production. Using the cloud to implement a DataOps approach, however, makes the entire data analytics process becomes much easier and faster.
Findings in Qubole’s State of DataOps report show that nearly six in 10 companies are currently using at least some cloud resources for big data processing, while 14 percent are running all big data processing in the cloud and 41 percent are running at least some data processing in the cloud.
“In many industries, insights are becoming a key differentiator - the faster you can use big data to make business decisions, the better. Companies want to deliver broad access to take advantage of these insights and the cloud has a clear advantage of time to insights, along with a hidden advantage,” Ashish noted. “Since these resources are available on demand, you can be more experimental in your approach. For example, if you are an executive and have a question, you can go to your data analyst and come back in an hour or a day with an answer. You can make your business move forward much faster than having to wait as you would with on-premises solutions.”
Big data management in the cloud also provides enterprises with a self-service platform that users can easily adapt to. This self-service data analytics model helps solve the problem of hiring more people with expertise in big data management.
According to Qubole’s State of DataOps report, 83 percent of respondents said their data teams are growing, yet 36 percent of these respondents expressed difficulty in hiring talent skilled in big data management. Qubole has provided both startup and enterprise customers with the benefit of keeping their engineering teams lean.
“At Lyft, our data volume has grown more than 100x in the past year alone, creating unique data infrastructure challenges for our team to solve. Qubole allows our business to continue scaling rapidly even as our infrastructure engineering team stays lean,” explained Chris Lambert, CTO of Lyft.
Mastering The Data-Driven Enterprise
Big data analytics drive today’s enterprise, yet understanding how to manage big data efficiently still remains a mystery for many organizations. Taking a DataOps approach allows organizations to manage all of their data in the cloud, automating the process. This provides users with a self-service model that results in a number of benefits such as quick and easy access to valuable analytics and the need to hire less big data experts.
"Having experienced this firsthand at Facebook, delivering on the promise of self-service access to data and analytics across the enterprise is extremely difficult and goes way beyond technology, involving rethinking processes, company culture and the operational model of the data team," Ashish said. "Until IT teams adopt a DataOps approach versus a more traditional command-and-control model, they'll remain a primary bottleneck to insights and their big data initiatives will continue to struggle. But there is a path -- some companies have successfully made the transformation, and others can learn from their experiences."