Co-founder and CEO of Qubole, the leading open data lake company for analytics and machine learning.
Getty
Most enterprises are using data analytics to gain competitive advantage, but the field of analytics continues to evolve, and streaming data has quickly emerged as the next big opportunity for innovation. But what is it, exactly, and how can CIOs put it to work in their organizations?
Streaming data — also called real-time data — is information that arrives continuously from various sources and is analyzed quickly so businesses can respond in near-real time to events on the ground. The data comes in from sources including websites, apps, imaging satellites, industrial sensors and other connected devices, and it can be put to work in a multitude of ways, which is partly what makes identifying use cases challenging.
IDC says nearly 30% of data generated by 2025 will be real-time in nature, and enterprises are using this data to build new products and improve operations and customer service. By 2022, Gartner, Inc. predicts that more than half of major new business applications will incorporate real-time data to improve decision-making.
This growth is due partly to the rapid emergence of a mature technology stack for ingesting and analyzing real-time data, including open-source technologies like Spark and Kafka. These are often used in conjunction with an open data lake, which makes it easy to collect all types of data and use it for machine learning and other advanced applications.
As you consider how to make use of streaming data, keep two things in mind:
1. Streaming data is rarely used in isolation. It's often combined with historical data to make it more useful, such as for identifying fraud, which is why data lakes are a good fit — they make it easy to combine different data types.
2. The value of real-time data depreciates with time. Whether you're looking at industrial telematics, social sentiment on Twitter or disease infection rates, the value of this data invariably diminishes the older it gets, so it's important to have a technology stack that allows applications to use data quickly before it gets old.
How can CIOs put this data to work in their own organizations? Here are several examples that illustrate some use cases and may spark ideas for your company — some common and some less so:
Increasing User Stickiness And In-App Purchases
Keeping users interested and engaged in apps and on websites is invaluable for increasing customer loyalty and retention. A gaming service like Roblox, for example, can observe players as they progress through a game and offer tips or "cheats" if they're stuck on a certain level.
Retailers can also make great use of real-time data to provide relevant offers and show related products that users might be interested in based on their current activity, location and other variables. These use cases all help to increase loyalty and drive engagement.
Transportation And Logistics
Boeing and other aircraft makers collect real-time flight data from jet engines and other onboard equipment to help airlines optimize fuel efficiency and reduce maintenance costs. The systems collect data during flights and can also monitor for wear and anomalies, speeding up maintenance and inspections upon landing.
Autonomous vehicles and advanced driver-assistance systems (ADAS) also lean on real-time data to understand surrounding traffic conditions. Away from public highways, streaming data is being used to manage freight operations and improve worker safety.
Fraud Detection
Detecting fraud goes beyond banks to any online service. In this use case, businesses can look at streaming data collected at the point of transaction — a website, ATM or point-of-sale device — and combine it with historical profile data to look for anomalies and other suspicious events that may indicate fraudulent activity.
Agriculture
Farming is an increasingly high-tech industry, and real-time data from multiple sources can be analyzed to help combat disease and improve yields. Sensors on farm equipment and in the soil monitor variables like moisture, nutrient levels and seed counts. This data can be combined with real-time weather data to help farmers determine optimal times for watering, fertilizing and harvesting.
Public Health
The pandemic has shown the need to monitor and predict the spread of disease patterns in real time and is one of the most critical uses of real-time data today. One healthcare company developed an app that tracks the emergence of new Covid-19 cases in real time by combining data from electronic health record systems. This data can also help health officials to identify surges and plan coordinated responses.
Outside of pandemics, data can be used to monitor flu outbreaks in real time, and companies are developing smart inhalers for asthma sufferers that will incorporate weather information to ensure patients take the right dose at the right time.
These examples only scrape the surface of how real-time data can be put to work. As 5G networks are built out and the internet of things continues to expand, the use cases for business will only expand. The primary purpose of streaming data is to understand what's happening in the present, but it can also be combined with historical data to turn insight into action. If you're not thinking about how to use streaming data today, you will want to very soon because it's likely your competitors already are.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
