TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The New Frontier in DataOps: Cloud Application Data

A new, more scalable approach to capturing, ingesting, and acting on SaaS application data is starting to emerge.

• By Joe Gaska
• February 2, 2021

As DataOps continues to fuel the silent arms race for historical data across every sector of our economy, organizations have turned their gaze to a new source of rich, highly structured information about their business velocity: SaaS application data.

For Further Reading: Adopting Agility for DataOps Best Practices for Data Management with DataOps Why Enterprises Are Turning to DataOps

It is estimated that 97 percent of organizations use cloud applications in their day-to-day operations, and most either inform or run their business inside these third-party apps. The historical data stored or continually overwritten in the applications has quickly become a high-value target for organizations' data consumers who are interested in knowing how and why revenue velocity has changed, how and why attrition or up-sell has changed, or how and where data corruption occurred.

Most answers to such questions are either informed or found in the SaaS applications where they occurred in the first place. The cause-and-effect patterns captured in your CRM, ERP, e-commerce, marketing automation, and other systems are a structured, time- series data set of changes that occur in your organization -- which makes them better at predicting business velocity changes than the traditional, monolithic notion of big data.

This is not news to most people. It's why some of the most iconic organizations in the world are rapidly trying to ingest and consume as much of their high-value SaaS application data as possible. What is new is how they are starting to do this.

If you need data stored in a SaaS application, you can likely get to parts of it through a combination of APIs or native integrations, but this approach doesn't scale well. APIs and integrations have limitations, may negatively impact cloud application performance, and may cause latency or even data corruption issues. They may simply be cost prohibitive -- think about the cost of increasing API call limits or in developing and maintaining real-time data pipelines for each SaaS application over time.

A new, more scalable approach to capturing, ingesting, and acting on SaaS application data is starting to emerge. Built on the key DataOps principle of self-organization, it makes historical SaaS application data broadly available for self-driven consumption within the organization. This approach is built on two critical architecture decisions: first, where SaaS application data is stored, and second, the fidelity and frequency with which it is captured.

Rather than hitting an application directly via APIs or even direct integrations, organizations are replicating their SaaS application data to their organization's cloud storage infrastructure (on Amazon Web Services, Microsoft Azure, or Google Cloud). Because traditional replication only provides a point-in-time snapshot of the data, enterprises are turning to SaaS data backup tools to capture a stream of historical change data instead of simply the latest state. What often remains in question is the frequency or fidelity with which the data consumers in an organization need to make use of the data.

The tradeoff between how much detail you want to know about changes in your business must be balanced with how much data you are willing to store, and how that stacks up against your competitor's willingness to do the same.