Preparing for the Elastic Age of BI and Analytics
New TDWI research uncovers three of the most significant drivers behind current and planned use of cloud computing for BI and analytics.
- By David Stodder
- October 10, 2016
We have entered the age of elastic computing. Are your strategies for business intelligence (BI), analytics, and data warehousing (DW) ready? Most business users and innovative IT leaders would likely answer that at least in terms of desire, they are more than ready.
Business users in particular have been waiting a long time for BI and DW systems that could be set up quickly and "elastically" grow or shrink as required by the business. They have been frustrated by traditional development and deployment of BI and DW systems, which involve long gestation periods and are typically built assuming fixed requirements for the volume and variety of data, kinds of reports, number of users, levels of concurrency, and so on.
With the advent of cloud computing, the era of standing up BI and DW systems once to last for all time may be coming to a close. However, cloud's elasticity brings new challenges as well as opportunities.
In October, TDWI will publish a new Best Practices Report, BI, Analytics, and the Cloud: Strategies for Business Agility, written by myself and Fern Halper, TDWI VP and senior research director for advanced analytics. Our research finds a strong uptick in both current and planned use of cloud computing for BI and analytics.
For example, half of our research survey participants said that their organizations are currently using some sort of cloud deployment model for analytics and nearly a third (31 percent) are planning to do so in the next few years. (Note: On October 12, Fern and I will present a webinar about the research and offer best practices recommendations.)
What's Driving the Cloud
Our research points to scalability, flexibility, and speed of development and deployment -- all qualities that define elasticity -- as the most significant drivers behind current and planned use of cloud computing for BI and analytics. Many cloud platform providers promise elasticity to appeal to market demand for systems that can be spun up and brought down at will, in some cases with the swipe of a credit card.
Today's enterprises want to bring up a system on demand, in the cloud, for any number of reasons, such as to track the performance of a new and unanticipated marketing or social media campaign, perform predictive analytics ahead of a product launch, crunch numbers at the end of a quarter, work through a simulated business scenario, or perform discovery analytics on a new source of data. Once done, they may want to scale down the system so they are not wasting money and computing resources.
Such scenarios would be difficult with on-premises systems that have been carefully configured for expected performance workloads that are assumed to go on indefinitely. It can be tough to reconfigure these systems for unexpected workloads.
Elasticity enabled by cloud computing therefore represents a major change in how organizations should think about BI, analytics, and DW systems. No longer do users' immediate requirements these systems need to be held hostage to long and costly development cycles, although organizations still need to do user requirements collection, data modeling, and cost-benefit analysis for the services they will be using.
Perhaps even more disruptive, however, is that no longer do individual users and departments necessarily have to wait for IT to set up systems. If a marketing department, for example, needs a BI system, why wait for IT's blessing and funding? This, of course, could lead to an explosion of "shadow IT" systems and data silos, something that will be a growing organizational challenge as cloud computing expands.
Open Source for BI and Analytics
Regarding data warehouses, already a significant number of organizations are creating data warehouses and data lakes in the cloud using either commercial software running in the cloud or open source technologies developed in the Apache Hadoop or Spark ecosystem. To that point, nearly half (48 percent) of research participants in the report survey said they are considering open source for BI and analytics and that the cloud could help them experiment with these technologies.
Organizations may not yet be deploying cloud-based systems to replace existing BI systems and data warehouses -- indeed, we found strong interest in "hybrid" architectures that incorporate both cloud and on-premises systems -- but for new systems, the cloud is attractive. A significant number are testing open source-based technologies for new systems.
Looking at front-end tools, our research finds that adoption of self-service BI and visual analytics is the most common current and planned use case for cloud analytics. This indicates that many companies are making use of software-as-a-service (SaaS) versions of these tools or are deploying BI and visual analytics servers not on premises but in public, private, or hybrid cloud platforms.
Our report details several user stories illustrating how organizations are choosing SaaS and cloud platforms to realize goals for higher flexibility and scalability -- and faster deployment of BI and analytics.
Avoiding the Pitfalls
Cloud computing will accelerate trends toward business-driven, self-service BI and analytics, enabling organizations to serve dynamic needs more easily by spinning up cloud-based data systems. Of course, there's never any free lunch; organizations need to be mindful of potential pitfalls.
Here are three issues that organizations should keep in mind:
Efficiency matters. Costs can rise with cloud computing if organizations are not efficient in using bandwidth and storage. Elasticity is often directly related to cost; some cloud providers' pricing escalates as systems scale out or more bandwidth is used. Efficient use of resources is also important to query performance in the same way that it is with on-premises systems.
Organizations should make sure that their cloud data management systems have adequate performance and optimization tools to monitor performance and overcome unevenness and inconsistency in the performance of individual nodes in parallel database clusters. Cloud data management systems also need to be resilient and available to handle the required number of concurrent users.
Update integration between BI, analytics, and business applications. Organizations can take advantage of modern APIs and microservices approaches to better integrate BI and analytics applications with the organization's CRM, financial forecasting, planning, and other business applications, which may also be moving to the cloud. Organizations should avoid making the gap between BI and analytics applications and other business applications even wider.
Moving to the cloud could enable organizations to update their integration technology. Some solution providers make it straightforward to embed BI and analytics routines into their applications. Cloud-based BI and analytics services within business applications could provide users with better visualization (such as dashboards) and more powerful analytics without having to switch to a different tool.
Data governance needs to be a priority. As architectures become more diverse, governance becomes harder. Organizations need to rewrite their governance models to include cloud platforms so that security, data ownership, and data stewardship over issues such as data quality and data lineage are addressed. Regulatory concerns about the use of data, which often prevent organizations from using public (but not necessarily private) cloud platforms, must also be satisfied. Make sure that business users and IT work together on governance models so that they will share responsibility for developing and executing governance policies.
The Future is Elastic
The elastic potential of cloud computing could make the BI, analytics, and DW landscape look very different in just a few short years. One can envision the majority of business users spinning up systems for dynamic needs, integrating BI and analytics services through APIs to their business applications, and using the scale-out potential of parallel computing to expand cloud platform systems as they need to work with more data -- and then shutting systems down when they're done.
To be successful and mindful of costs, organizations need to be smart about how they use resources and govern the emerging environment with even more care than they do on-premises systems.
About the Author
David Stodder is director of TDWI Research for business intelligence. He focuses on providing research-based insight and best practices for organizations implementing BI, analytics, performance management, data discovery, data visualization, and related technologies and methods. He is the author of TDWI Best Practices Reports on mobile BI and customer analytics in the age of social media, as well as TDWI Checklist Reports on data discovery and information management. He has chaired TDWI conferences on BI agility and big data analytics. Stodder has provided thought leadership on BI, information management, and IT management for over two decades. He has served as vice president and research director with Ventana Research, and he was the founding chief editor of Intelligent Enterprise, where he served as editorial director for nine years.