TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Trustworthy Data: The Goal of Data Quality and Governance

Is poor data governance and slow data prep really a problem? It is when it erodes confidence in the quality of your data.

• By Lindsay Stares
• December 8, 2017

It's common today to say that analysts spend too much time preparing data and not enough analyzing it. Most recently, a new study from TMMData and the Digital Analytics Association finds that "nearly 40% of data professionals (37.5%) spend more than 20 hours per week accessing, blending, and preparing data rather than performing actual analysis."

On the surface, this problem mostly concerns the valuable time of analysts -- and their better-paid cousins, the data scientists -- but data preparation challenges don't exist in a vacuum. Data that's difficult to access or integrate can lead to poor data quality, and untrustworthy data has serious implications for many industries.

The TMMData survey report put it baldly: "Data professionals lack confidence in data accuracy, which poses an existential threat to the industry."

Quality Issues in the Data Industry

Whether it's because of preparation problems, access issues, a lack of control, or other factors, trust in data is a widespread concern. According to the TMMData survey, almost half of respondents (48 percent) "question the accuracy" of the data they work with.

When the respondents were split between analysts and managers, 5 percent of leaders and managers believed their data quality was excellent, but only 3 percent of analysts and staff felt the same. Meanwhile, 11 percent of staff and 12 percent of managers reported their data quality was only fair or poor.

Other surveys have found leaders with even less trust in their data. According to Experian Data Quality's 2017 global data management benchmark report, on average, C-level executives believe that "33 percent of their organizations' data is inaccurate." Furthermore, "nearly half of organizations globally say that a lack of trust in their data contributes to increased risk of non-compliance and regulatory penalties (52%) and a downturn in customer loyalty (51%)."

It isn't just a problem for data companies, of course. A survey from Bazaarvoice and Advertising Age on data use in digital marketing found that most marketers didn't understand the sources of the data they received. Seventy-five percent of marketers weren't fully confident that the data they were using was effective in reaching their target consumers.

The problem isn't limited to large enterprises, either. Jennifer Moore is the senior director of marketing and marketing strategy at TMMData and an author of the whitepaper "State of Digital Analytics: The Persistent Challenge of Data Access & Governance."

"When we embarked upon this study," she told Upside, "we expected to see differences among organizations of varying sizes, but our findings revealed that data access and quality is a challenge for organizations regardless of size. There was an expectation that large companies might apply more resources to set them ahead of the curve, or that small companies would have fewer departmental siloes to centralize, but instead our research found that organizations of any size struggle with these data challenges."

What Can Data Enterprises Do?

Whether you use data to drive your own business or sell data as a product, you can see the importance (and unfortunate rarity) of having data you can trust. Here are a few principles to increase confidence in your data.

Transparency is essential, especially for enterprises that are selling data to a third party. In the marketing study, "only 29% of agency respondents and 26% of brand marketers reported being 'extremely/very clear' when it comes to where their data comes from." Both groups of marketers are seeking data providers that will "step up and prove their worth through greater clarity and transparency."

Better data prep skills and processes can lead to data that is trusted. Seventy percent of analysts in the TMMData survey were taking it upon themselves to learn new programming skills specifically to improve data access and preparation, and more than 25 percent of those had spent 80 hours or more on the project. That survey also found that the second most popular data-related investment priority for next year is professional development. If you need higher quality data, invest in your employees as well as your tools.

Improve your governance program. More than half of the respondents to the Experian study (51 percent) said their data governance was ineffective. Less than a quarter of respondents to the TMMData survey had an active governance program in place. At the same time, organizations are moving forward. Nearly a third in the TMMData survey are "planning or researching a formalized data governance program," and "nearly 20% are in the early stages of a roll out."

As Moore points out, "The research suggests that organizations implement two key best practices: first, develop a formal data governance system, and second, identify the best tools to support their data prep, integration, and management needs. Governance at the organization level will have a substantial impact on data quality, while self-service data management tools can drastically free up analysts' time for higher-level business analysis and strategy.

"Because the survey responses cited cultural and political challenges around data governance and access, our hope is that this research will help analytics-focused units develop strong business cases for investments like governance, integration, and automation tools. This research is very much a benchmarking resource, so perhaps seeing that other organizations are prioritizing investments in integration resources, professional development, and new hiring will help motivate leadership to act."

No Simple Solution

There's no quick fix for creating trustworthy data. You need a combination of employee diligence, tools for data management, governance, and cleansing, and a growing track record of success to bring executives on board. However, as your employees and managers gain confidence in the processes and the data itself, you'll see better results from your analytics and more loyalty from your customers.