Top Twelve Priorities for Data Warehouse Modernization
Top Twelve Priorities for Data Warehouse Modernization
By Philip Russom, Senior Research Director for Data Management, TDWI
No matter the vintage or sophistication of your organization’s data warehouse (DW) and the environment around it, it probably needs to be modernized in one or more ways. That’s because DWs and requirements for them continue to evolve. Many users need to get caught up by realigning the DW environment with new business requirements and technology challenges. Once caught up, they need a strategy for continuous modernization.
To help you organize your modernization efforts, here’s a list of the top twelve priorities for data warehouse modernization, including a few comments about why these are important. Think of the priorities as recommendations, requirements, or rules that can guide user organizations into successful strategies for implementing a modernization project.
1. Embrace change. Data warehouse modernization is real; a recent TDWI survey says that 76% of DWs are evolving moderately or dramatically. Given the rampant amount of change in markets and individual businesses, it’s unlikely the status quo will serve you and your organization for much longer. Besides, change is an opportunity for improvement, as long as you manage it with specific directions in mind.
2. Make realignment with business goals your top priority. This is the leading driver according to a recent TDWI survey. Learn the goals of the business and collaborate with business and technical people to determine how business goals map to technology and data. Then base your modernizations on the requirements thus defined. If alignment is achieved, the whole business will modernize, not just the warehouse. And that’s the real point.
3. Make DW capacity a high priority on the technology side. The second most pressing driver is greater capacity for growing data, users, reports. This is no surprise given the explosive growth of traditional enterprise data and new big data. 3-10TB is today’s norm for DW data volume in the average-size organization; however, the norm will soon become 10-100TB, as DW programs graduate from lesser data volumes to greater ones. These are known capacity goals for successful DWs, so keep them in mind when planning capacity modernization.
4. Make analytics a priority, too. One third of DW professionals modernize for better and newer analytics. That’s a technology challenge for the warehouse, since diverse analytic techniques have diverse data preparation requirements, and they don’t all fit the traditional warehouse. Therefore, additional data platforms and tools that complement older ones may be in order. Keep in mind that analytics is what business users want; your pristine data and elegant architecture won’t mean much, if modernization fails to deliver relevant analytics.
5. Don’t forget the related systems and disciplines that also need modernization. Top priorities are analytics, reporting, and data integration, followed by development methods and team characteristics. Align the modernization of the DW, so it can ably provision the data in a manner that these other disciplines require for their success.
6. Don’t be seduced by new, shiny objects. There are lots of new and cool technologies and tools available today, and many get evaluated for DW modernization. Before adopting one, be sure it goes beyond the bling to satisfy real-world requirements in a performant and cost-effective manner.
7. Assume that you’ll need multiple manifestations of modernization. To get the desired results, you should consider multiple modernization strategies, but try not to execute them all at once, in a big bang.
8. Be familiar with today’s tools and techniques for the modern data warehouse environment (DWE). Extending the number and type of standalone platforms within a DWE is one of the strongest trends in data warehouse modernization, because it adds value in the form of additional platforms, without ripping out or replacing established platforms.
9. Adjust the large-scale architecture of your DWE. The rise of the multi-platform DWE is forcing the modernization of system architectures. For most situations, you will keep and improve your centralized, relational DW. But you should expect to complement it with other platforms, then migrate data and balance workloads among platforms. This requires you to rework the large-scale architecture, which determines how diverse platforms integrate and interoperate, plus which data goes where and how data show flow among platforms.
10. Reevaluate your DW platform. The condition of your data is important, but it’s all for naught if the platform can’t capture, manage, and deliver data with speed, scale, and broad functionality at a reasonable cost. Replacing a DW platform is disruptive and expensive for a business. Therefore, consider leaving your existing DW platform in place, but update it and complement it with other systems. Even so, grossly deficient or outmoded platforms should be replaced.
11. Consider Hadoop for various roles in the DWE. Hadoop’s massive and cheap storage offloads older systems by taking responsibility for data staging, ELT push down, and the archiving of detailed source data (retained for advanced analytics). Hadoop also serves as a massively parallel execution engine for a wide variety of set-based and algorithmic analytic methods. Conventional wisdom says Hadoop usually complements a DW without replacing it. That’s what early adaptors do with Hadoop in DWEs today. And the number of organizations integrating Hadoop with a DW continues to increase.
12. Develop plans and recurring cycles for DW modernization. Most DW teams have settled on a quarterly schedule for updating DWs. This applies to tasks of many sizes; well-contained phases of some modernization projects may fit this scheme, as well. However, large-scale modernizations typically need their own plan. The more disruptive a modernization (such as rip-and-replace), the more critical to success is the multi-phase plan (sometimes the multi-year plan). Modernization affects business users and their processes; for minimal disruption, business managers should be involved in developing and executing modernization plans.
ANNOUNCEMENT
To learn more about modernizing data warehouses and related IT systems, attend my TDWI webinar Data Warehouse Modernization in the age of Big Data Analytics, coming up on April 14, 2016. Register online for the webinar: http://bit.ly/DWMod16
This webinar will quantify trends in data warehouse modernization and catalog technologies that are relevant. It will also document strategies and user best practices for organizing modernization projects. The goal is to help DW professionals and their business counterparts plan the next generation of their data warehouse, in alignment with business goals.
Posted on March 24, 2016