The Benefits of Streaming Data Are Contagious
Enterprises' desire to access real-time data spreads quickly once the right use cases are found.
Data streaming's always been with us, but now enterprises are suddenly getting hip to its value.
The term "streaming" is shorthand for real-time data feeds, be they application event traffic, messages or events generated by sensors and devices, or continuous flows of data.
For decades, enterprise data management has been organized around a data-at-rest paradigm: a model in which data is processed in discrete batch intervals. Streaming data is one of several pieces of a data-in-motion paradigm in which data is ingested more or less continuously.
The enterprise environments of yore did not lack for potential streaming sources. There was the venerable enterprise service bus (ESB) -- backbone for application event traffic and locus of enterprise application integration (EAI) efforts -- for starters. Machine-generated events -- e.g., message traffic created by control systems and other sensor-equipped devices/equipment -- aren't new, either.
However, in the past enterprises weren't overly interested in this data. They didn't bother collecting it. That has changed, says Steve Wilkes, cofounder and CTO with Striim, a provider of streaming analytics and integration software.
Streaming on a Use-Case Basis
"Enterprises are doing [streaming] on a use-case basis. The thing that has happened over the last two years ... is [that] people are understanding what streaming means, what streaming data means, what real time means, and they're now identifying use cases across lots of different industries where that is really important," Wilkes says, stressing that enterprises are using streaming-analytics technologies to complement traditional data management and decision support practices.
"It is not that enterprises will say, 'Tomorrow we're going to move everything to a streaming architecture.' Nobody is saying that. It's rather 'We've identified a use case that we'll move forward with,'" he explains. In a growing number of cases, Wilkes notes, enterprises are using streaming technologies to continuously feed DBMSs. This permits them to support real-time applications.
"If it's the database [use case], we'll use change data capture [technology] ... but the idea is that we can make this changed data available to different [constituencies] -- maybe people want to do real-time analytics on it, maybe we'll also push it out to Kafka," he says.
Potential and Challenges
Streaming's data-in-motion paradigm can help accelerate data delivery. It's one of several technologies -- data virtualization is another, and representational state transfer (REST) still another -- that have the potential to knit an enterprise together via low-latency data exchange.
Wilkes isn't quite that idealistic, however. He points out that even as streaming enables new (near-real-time or low-latency) use cases, it also presents new challenges.
He uses the example of streaming data to cloud services. "If we're pushing data into the cloud, maybe we need to obfuscate it a bit, because we don't want sensitive data going out to the cloud. That's something we can do [in Striim] pretty easily," he comments.
A Contagious Transformation
In a sense, the logic of streaming is viral: once an enterprise gets comfortable with it, many long-established batch-based processes will be upgraded. "It's a slow transition where some of the things that were done in batch start to become available in streaming," Wilkes says.
"Right now, a lot of the enterprise stuff is around data movement and data prep, [as well as] integration of enterprise to cloud, but [enterprises are] starting to realize that you can move some of the analytics that used to be done after the fact into a real-time architecture. That's infectious."