Achieving Scalable, Agile, and Comprehensive Data Management and Governance (Part 1 of 3)
David Stodder, senior research director for business intelligence with TDWI, discusses his newest Best Practices Report, which covers how to achieve the best in data management and governance.
- By Upside Staff
- October 6, 2023
In this recent “Speaking of Data” podcast, TDWI’s Dave Stodder discussed the latest developments in data management and governance. Stodder is senior research director for business intelligence at TDWI. [Editor’s note: Speaker quotations have been edited for length and clarity.]
“Agile and scalable data management and governance are core topics at TDWI,” Stodder began. “Today’s data environments involve aspects such as on-premises data, hybrid clouds, multiple data silos, and many other complex situations, often built up over time in a piecemeal fashion.” As a result, getting a unified picture of their data is a real challenge for organizations, he said, as is finding ways to put it to use to reduce costs, eliminate redundancy, and improve data quality.
“The key,” he added, “is balancing stability and flexibility in your data operations so that you’re keeping pace with business demands and responding to unanticipated events while still effectively governing your data to ensure your efforts are producing value.” Stodder pointed out that one recent improvement in this area is that the business is now the main driver behind the move to the cloud, which has helped bridge the historical gaps between the business and IT.
However, Stodder continued, organizations are having only isolated success with these goals, according to research conducted for the writing of this report. For example, he said, only 8% of respondents said they are highly successful currently and confident about being able to handle future challenges.
AI is also a top concern for survey respondents, with 43% saying AI/ML development (which includes streamlining data preparation and improving data availability) is their main goal. However, Stodder added that in the course of writing the report he saw that many organizations are still quite fragmented in their handling of AI practices, such as model governance and monitoring.
“Data governance in general is fairly uneven,” he explained. “In terms of protecting sensitive data, there’s been improvement, though. Organizations have been more willing to shut down risky programs that may expose sensitive data even at the expense of losing competitive advantage rather than run afoul of regulations.” As a sign of this improvement, he added, 73% of survey respondents said they were at least somewhat successful at meeting their regulatory and compliance objectives.
Another key concern Stodder discussed was the highly distributed nature of today’s data environment.
“Creating data silos goes hand in hand with data democratization,” he said. “Forty-one percent of our survey respondents said managing data silos was one of their top three challenges.” To address this, he said, many are turning to solutions such as data virtualization, data fabrics, or data meshes. He also added that the research showed roughly 30% as already using data virtualization and about the same number planning to.
Stodder explained another popular solution: the data lake house -- a combination of the structured nature of the data warehouse and the open nature of the data lake -- which 46% of survey respondents said they were planning to implement.
“At the center of many of these approaches,” he continued, “is the data catalog -- a storage tool for all kinds of metadata and knowledge about the data.” A primary benefit to many of these tools and approaches, Stodder said, is the ability to access data without a lot of data movement, which is a big concern for roughly one-third of survey respondents.
“In the end, the report’s main recommendation is to stay on top of modern technologies and practices to improve data quality.” That may seem obvious, he explained, and it’s easy to sleep on that issue, but it’s a problem that’s never really “solved.” Once you think you have everything nailed down, he said, it changes. It requires a lot of time and attention from the whole range of stakeholders.
Another essential aspect Stodder mentioned was the role of data stewardship in managing many of the data issues organizations face, from governance to improving data quality. Many stewards also have other positions they’re filling, he said, so the degree to which the tasks of the data steward can be automated can prove very valuable.