What’s Ahead in 2024 with TDWI’s Fern Halper and Dave Stodder
TDWI analysts Fern Halper, Ph.D., and David Stodder discuss what’s ahead for the data and analytics industry in 2024, including data integration modernization, generative AI, and self-service capabilities.
- By Upside Staff
- January 5, 2024
In this recent “Speaking of Data” podcast, TDWI’s Fern Halper and David Stodder discussed what’s ahead for the data and analytics industry in 2024, including data integration modernization, generative AI, and self-service capabilities. Halper is VP and senior research director for advanced analytics, and Stodder is senior research director for business intelligence. [Editor’s note: Speaker quotations have been edited for length and clarity.]
The conversation began with Stodder laying out his view of where data integration is headed in 2024.
“The whole data integration field is under tremendous pressure from applications designed to run 24/7, in addition to trying to support AI, machine learning, and generative AI applications,” he said. “Organizations have moved to the cloud to get away from the more rigid structures of legacy systems, but this has brought its own issues. Any time you move data pipelines and ETL to a new platform, you have to make sure you’re retaining enough of the prior knowledge about the data and what people are using it for.”
The goal, he explained, is to gain agility and flexibility, and to be able to personalize your systems to the needs of users. To achieve that end, AI and automation are being incorporated into data integration tools and platforms, which also helps address data quality issues that continue to plague organizations.
Halper added that these data quality issues will only be compounded as companies incorporate more diverse semistructured and unstructured data types to feed their AI systems. “For example,” she said, “what does it mean to have ‘high-quality’ unstructured data?” She expects this to be a big question for 2024.
Continuing on about AI -- and specifically generative AI -- Halper predicted the major development will be organizations developing their own in-house generative AI solutions using their own proprietary data -- both to address security concerns and to help differentiate their solutions from the others on the market.
“A good example is in our own marketing department,” she said. “We’re using ML/AI and generative AI to craft personalized messages based on customer data to enhance the effectiveness of our campaigns. We have all our tools for segmenting and ranking our customers, and once that’s done, we can use generative AI to create personalized message content to the top customers in each segment using our own proprietary data.”
Halper also noted that this evolution would likely require an accompanying evolution in tech teams. “We’re accustomed to data scientists building AI models, software developers developing applications, and data engineers dealing with the data,” she said, “but going forward they’re going to have to come together to develop this next generation of AI applications -- which is not something they’ve done well historically.”
She also brought up the idea of new tuning techniques, such as retrieval augmented generation (RAG), and new pieces of the data infrastructure, such as vector databases. These developments will help the move towards generative AI to continue.
Halper and Stodder agreed that these innovations will also help further self-service data access in the coming year. Stodder anticipates a great deal of improvement in helping users write their own natural language queries with the help of generative AI -- a copilot, as it were, that will suggest possible queries based on the user’s persona and history of usage.
The conversation then turned to the question of governance and how it will affect new efforts in machine learning, artificial intelligence, and generative AI.
Halper explained that TDWI research consistently finds that governance is always a top priority, as well as a top challenge. “With the addition of new data types and analytics models, there’s just that much more to govern. However, in 2024, I expect to see the priorities for governance to shift somewhat. We did a Best Practices Report about a year ago about responsible data and analytics, and our research showed that respondents weren’t as concerned about matters of ethics, bias, and explainability at the time. However, with the legislative discussions that took place in 2023 -- such as the Algorithmic Accountability Act in the U.S. and the EU AI Act in Europe -- we should expect to see that change.”
The conversation closed with a discussion of how to get -- and keep -- the buy-in of your executive leadership for data initiatives in the coming year.
“Technology is changing so fast, it’s critical to think about the leadership issues that might arise,” Stodder warned. “Given that, it’s essential to always be thinking about what the outcomes are that you’re trying to achieve and how they relate to the business.” For the data teams, this means being always up to date about the data. “The whole range of tools, such as data catalogs and lineage tools, business glossaries, and so on, will help keep people engaged.”