New TDWI Assessment Examines the State of Data Quality Maturity Today
Fern Halper, Ph.D., vice president and senior research director for advanced analytics at TDWI, discusses modern data quality and a new TDWI tool for assessing its maturity.
- By Upside Staff
- March 8, 2024
In this “Speaking of Data” podcast, Fern Halper, Ph.D., spoke about factors affecting data quality today and introduced a new tool for organizations to assess the maturity of their implementations. Halper is vice president and senior research director for advanced analytics at TDWI. [Editor’s note: Speaker quotations have been edited for length and clarity.]
“With data becoming such a critical part of a business’s ability to compete, it’s no wonder there’s a growing emphasis on data quality,” Halper began. “Organizations need better and faster insights in order to succeed, and for that they need better, more enriched data sets for advanced analytics -- such as predictive analytics and machine learning.”
She explained that to do this, organizations are not only increasing the amount of traditional, structured data they’re collecting, they’re also looking for newer data types, such as unstructured text data or semistructured data from websites. Taken together, these various types of data can offer significantly more opportunities for insights, she added.
As an example, Halper mentioned the idea of an organization using notes from its call center -- typically unstructured or semistructured text data -- to analyze customer satisfaction, either with a particular product or with the company as a whole. This information can then be fed back into an analytics or machine learning routine and reveal patterns or other insights meaningful to the company.
“Regardless of the type of data or its end use,” she said, “the original data must be high quality. It must be accurate, complete, timely, trustworthy, and fit for purpose.”
One issue with this, she noted, is that the concept of what constitutes “high quality” is not always so obvious.
“For instance, what is ‘high-quality image data’? What constitutes high quality for unstructured text?” she said. “These are questions that most organizations haven’t thought about yet, given how new the data types they relate to are.” This is on top of the fact that most organizations had never really solved these questions as they related to the traditional data they were already collecting.
She said that a significant reason for releasing the Data Quality Maturity Assessment is to help assess where organizations truly stand. For example, in a recent TDWI survey, less than 50% of respondents reported being satisfied with the quality of their data.
“Data quality touches on so many other aspects of an organization’s welfare,” Halper said. “For instance, it’s an important part of data governance -- something our research shows is often at the top of the list of priorities for data management.”
She went on to explain the five dimensions of the Data Quality Maturity Assessment model in greater detail.
- Organizational commitment: Is there a strategy in place throughout the organization? Is there awareness? Is there funding to maintain data quality practices?
- Roles and responsibilities: Are there people in place who are accountable for data quality? Is there training available?
- Data quality management: What are the tools and processes in place to make sure that data is accurate, reliable, relevant, timely, and so on? What processes are in place for exposing poor-quality data and remediating it?
- Assurance and impact: How well are data quality processes working? Is the organization actually measuring data quality?
- Tools: What tools are in place? How advanced are they? Do they use automation or augmentation to help manage the tasks involved in data quality?
“There’s also an accompanying guide that will help make sense of the assessment questions and provide tips for how to improve in the various dimensions of the model,” she noted.
“Another thing to be on the lookout for,” Halper added, “is that we will be publishing a comprehensive ‘State of Data Quality Maturity’ report next quarter that will incorporate the results of the assessment, as well as a number of other sources.”
To see more TDWI maturity models and assessments, visit TDWI.org.