Big Data Challenges and Opportunity

It’s easy to get caught up in the hype and opportunity of big data. However, one of the reasons big data is so underutilized is because big data and big data technologies also present many challenges. One survey found that 55% of big data projects are never completed. This finding was repeated in a second survey, that found the majority of on-premises big data projects aren’t successful. So what’s the problem with big data?

Top Big Data Challenges

Hadoop is Hard

While Hadoop and the surrounding ecosystem of tools is lauded for its ability to handle massive volumes of structured and unstructured data, the software isn’t easy to manage or use. Since the technology is relatively new, many data professionals aren’t familiar with how to manage Hadoop. Add to that the fact that Hadoop frequently requires extensive internal resources to maintain, and many companies are left devoting most of their resources to the technology rather than to the actual big data problem they are trying to solve. In the survey mentioned above, 73% of respondents claimed understanding the big data platform was the most significant challenge of a big data project. Read More.

Download Big Data Success Sheet


With big data, it’s crucial to be able to scale up and down on-demand. Many organizations fail to take into account how quickly a big data project can grow and evolve. Constantly pausing a project to add additional resources cuts into time for data analysis. Big data workloads also tend to be bursty, making it difficult to predict where resources should be allocated. The extent of this big data challenge varies by solution. A solution in the cloud will scale much easier and faster than an on-premises solution. Read More.

Lack of Talent

Businesses are feeling the data talent shortage. Not only is there a shortage of data scientists, but to successfully implement a big data project requires a sophisticated team of developers, data scientists and analysts who also have a sufficient amount of domain knowledge to identify valuable insights. Many big data vendors seek to overcome this big data challenge by providing their own educational resources or by providing the bulk of the management.


Having more data doesn’t necessarily lead to actionable insights. A key challenge for data science teams is to identify a clear business objective and the appropriate data sources to collect and analyze to meet that objective. The challenge doesn’t stop there, however. Once key patterns have been identified, businesses must be prepared to act and make necessary changes in order to derive business value from them. Read More.


Data quality is not a new concern, but the ability to store every piece of data a business produces in its original form compounds the problem. Dirty data costs companies in the United States $600 billion every year. Common causes of dirty data that must be addressed include user input errors, duplicate data and incorrect data linking. In addition to being meticulous at maintaining and cleaning data, big data algorithms can also be used to help clean data. Read More.


Keeping that vast lake of data secure is another big data challenge. Specific challenges include:

  1. User authentication for every team and team member accessing the data.
  2. Restricting access based on a user’s need.
  3. Recording data access histories and meeting other compliance regulations
  4. Proper use of encryption on data in-transit and at rest.

Read More


It’s difficult to project the cost of a big data project, and given how quickly they scale, can quickly eat up resources. The challenge lies in taking into account all costs of the project from acquiring new hardware, to paying a cloud provider, to hiring additional personnel. Businesses pursuing on-premises projects must remember the cost of training, maintenance and expansion. Big data in the cloud projects must carefully evaluate the service-level agreement with the provider to determine how usage will be billed and if there will be any additional fees.


While the number of big data challenges can be overwhelming, it also presents an opportunity. Those businesses who are able to identify the right infrastructure for their big data project and follow best practices for implementation will see a significant competitive advantage. Entrepreneurs have also capitalized on big data technology to create new products and services.