TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Evolving Data Warehouse Architectures: An Overview in 35 Tweets

By Philip Russom
Research Director for Data Management, TDWI

To help you better understand the ongoing evolution of data warehouse architectures and why you should care, I’d like to share with you the series of 35 tweets I recently issued on the topic. I think you’ll find the tweets interesting because they provide an overview of big data management and its best practices in a form that’s compact, yet amazingly comprehensive.

Every tweet I wrote was a short sound bite or stat bite drawn from my recent TDWI report Evolving Data Warehouse Architectures in the Age of Big Data. Many of the tweets focus on a statistic cited in the report, while other tweets are definitions stated in the report.

I left in the arcane acronyms, abbreviations, and incomplete sentences typical of tweets, because I think that all of you already know them or can figure them out. Even so, I deleted a few tiny URLs, hashtags, and repetitive phrases. I issued the tweets in groups, on related topics; so I’ve added some headings to this blog to show that organization. Otherwise, these are raw tweets.

Basic Components of the Average Data Warehouse Architecture

1. Most DW Arch’s have 4 layers: logical, physical, hardware topology, data standards.
2. DW logical architecture is mostly about data models, entity models & relationships.
3. DW logical arch also defines standards for data models, dev practices, interfaces, etc.
4. DW physical architecture is mostly a plan for data deployment on servers.
5. DW physical arch also defines topology for hardware & software servers plus interfaces.

Users’ Views of Architectural Components

6. #TDWI SURVEY SEZ: Data standards & rules are highest priority (71%) of #EDW architecture.
7. #TDWI SURVEY SEZ: Logical design (66%) is the starting point of an #EDW architecture.
8. #TDWI SURVEY SEZ: Physical plan (56%) locates logical pieces in an #EDW architecture.
9. #TDWI SURVEY SEZ: Only 12% have #EDW that’s “collection of data & platforms without a plan.”
10. #TDWI SURVEY SEZ: Only 12% feel Inmon vs Kimball argument is priority for #EDW architecture.

The Evolution of Data Warehouse Architectures

11. #TDWI SURVEY SEZ: 79% say their #DataWarehouse has an architecture.
12. #TDWI SURVEY SEZ: #EDW arch is evolving dramatically (22%), moderately (54%) or slightly (22%)
13. #TDWI SURVEY SEZ: Driving #EDW arch evolution: #Analytics 57%, #BigData 56%, #RealTime 41%.
14. #TDWI SURVEY SEZ: Driving #EDW arch evolution: BizPerfMgt 38%, OLAP 30%, UnstrucData 25%.
15. #TDWI SURVEY SEZ: Driving #EDW arch evolution: competition 45%, compliance 29%, dep’ts 29%.

The Importance of Data Warehouse Architectures

16. #TDWI SURVEY SEZ: Architecture extremely (79%) or moderately (19%) important to #EDW success.
17. #TDWI SURVEY SEZ: #EDW Architecture is an opportunity (84%), not a problem (16%).

Benefits and Barriers for Data Warehouse Architecture

18. #TDWI SURVEY SEZ: Stuff that benefits from #DWarch: #analytics, biz value, data breadth.
19. #TDWI SURVEY SEZ: Barriers to #DWarch success: skills gap, sponsorship, #DataMgt, funding.

Multi-Platform Data Warehouse Environments

20. #EDWarch trend: more standalone platforms: #analytics DBMSs, columnar, appliances, #Hadoop, etc.
21. As #EDW workloads get more diverse, so do types of standalone data platforms in #EDW environment.
22. As types and numbers of data platforms grow in DW environs, architecture gets ever more distributed. #
23. Distributed #EDWarch is good&bad: provides workload optimized platforms. But may spawn data silos.
24. Logical layer of #EDWarch more important than ever to unite big design across multi data platforms.

Single-Platform versus Multi-Platform DW Architectures

25. #TDWI SURVEY SEZ: Totally pure #EDWarchs are rare. Only 15% have central monolithic #EDW.
26. #TDWI SURVEY SEZ: Hybrid #EDWarchs are most common today = central #EDW + a few other data platforms (37%).
27. #TDWI SURVEY SEZ: 2nd most common Hybrid #EDWarch = central #EDW + many other data platforms (16%).
28. #TDWI SURVEY SEZ: Sometimes #EDW plays small role in #EDWarch compared to workload platforms (15%).
29. #TDWI SURVEY SEZ: Some organizations (15%) have many workload-specific data platforms, but no true DW.

Big Data’s Influence on Evolving DW Architectures

30. #TDWI SURVEY SEZ: 41% will extend existing core #EDW to handle #BigData.
31. #TDWI SURVEY SEZ: 25% will deploy new data platforms to handle #BigData.
32. #TDWI SURVEY SEZ: 23% have no strategy for their #EDW’s architecture, though they need one.
33. #TDWI SURVEY SEZ: Only 6% feel they don’t need a strategy for their #EDW’s architecture.

Reports and Analytics have Different DW Architecture Needs

34. Many users preserve #EDW for reporting, BizPerfMgt & OLAP, but take #analytics data elsewhere.
35. Data prep for reports differs from same for #analytics. So, many users prep data on separate platforms.

Want to learn more about evolving data warehouse architectures?

For a more detailed discussion—in a traditional publication!—get the TDWI Best Practices Report, titled Evolving Data Warehouse Architectures in the Age of Big Data, which is available in a PDF file via a free download.

You can also register for and replay my TDWI Webinar, where I present the findings of the TDWI report Evolving Data Warehouse Architectures in the Age of Big Data.

Posted by Philip Russom, Ph.D. on April 15, 2014