TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Real-Time Analytics: Challenges and Solutions

Four important challenges your enterprise may encounter when adopting real-time analytics and suggestions for overcoming them.

• By Irene Makaranka
• June 15, 2018

As a data analytics researcher, I know that implementing real-time analytics is a huge task for most enterprises, especially for those dealing with big data. It's worth the work, though; real-time analytics can help your enterprise reach insight faster and handle streaming data sources that give your analysis more depth. I've identified four key challenges and key tactics to help you overcome them.

For Further Reading: Reach Real-time Analytics on the Data Lake with GPU Acceleration Enabling the Real-Time Enterprise with Data Streaming Big Data, Infrastructure, and Performance

Challenge #1: A vague definition of real time

In our data consulting practice, our clients have different interpretations of the term real time. In the context of analytics, some believe real time means getting instantaneous insights and others are fine waiting several minutes between data collection and the analytics system's response.

Different interpretations result in inconsistent requirements. Imagine that the C-suite has opted to adopt real-time analytics but the management team understands the term in a different way and has different expectations. Will such a project be successful? Probably not. A vague definition means uncertainty about possible use cases and which business tasks (both current and future) can be solved.

Key takeaway: You must invest significant time and effort to gather detailed requirements from all stakeholders. At the end of this stage, your team must unanimously agree on what real time means, what data you need in real time, and what data sources you should use.

Challenge #2: An irrelevant architecture

Once you've nailed down the meaning of real time and clearly formulated the requirements for real-time analytics, it's time to proceed with the architecture design. Your architecture will need the ability to process your data at high speed. However, the processing-speed requirements can vary from milliseconds to minutes, depending on the data source and application.

Your architecture should also be able to deal with spikes in data volume and be able to scale up as your data grows. Certainly, you don't want to find out that the architecture that seemed perfect a year or two ago is not able to process your data volume if it doubles.

Companies that plan to adopt real-time analytics often lose sight of offline analytics, but you need both real-time and offline analytics to get insights. For example, sending instant alerts is a great application for real-time analytics; identifying models and patterns with machine learning is a time-consuming process not suitable for real-time processing.

Running real-time analytics and offline analytics on the same data may create conflicts for computing resources and hinder performance. Your architecture should be designed to resolve such conflicts.

To begin, examine sample architectures for a real-time analytics solution to trigger your own ideas. [Editor's note: there's a sample illustration in the "A typical architecture for real-time big data analytics" section of this article.] Alternatively, you may prefer to hire a professional consultant to design an architecture tailored to your needs.

Key takeaway: A well-designed architecture is a critical success factor, so pay special attention to the fundamental architectural issues. A typical architecture is a good foundation, but it can be further tailored to achieve better performance.

Challenge #3: Adjusting internal processes

Adopting real-time analytics includes such important activities as gathering requirements, designing the solution's architecture, choosing the right technology stack, and solving hardware and software issues. However, because of these technical tasks, enterprises often overlook the question of what they should do with their internal processes.

You're implementing real-time analytics for a reason, so you aren't satisfied with how your internal processes had been running. For example, perhaps you are a manufacturer that isn't happy with your equipment repair time. Breakdowns are always unexpected, and your maintenance team might spend hours identifying a cause, only to be unable to fix the machine because they don't have the replacement part.

With real-time analytics, your requirements for the maintenance team's operation will definitely change. You will expect faster fixes as well as preventive maintenance based on the data gathered. To make the most of the analytics solution, you should revise existing maintenance processes, key performance indicators, and job descriptions.

Key takeaway: Your enterprise can maximize the benefits of real-time analytics if you do not regard it as your ultimate goal but only as a helpful instrument and a starting point for improving your internal processes.

Challenge #4: Employees' resistance to change

If you are used to traditional business intelligence supporting your company's operations, implementing real-time analytics is a huge step forward. However, you should not let this change become disruptive.

The upgrade of your analytics system should not be frightening. On the contrary, it opens up new opportunities. Prepare your staff to embrace them. If your team used to receive daily or weekly reports built mainly on internal data sources, your new solution will make a whole array of new information available. For instance, instead of analyzing occupancy rate reports, a hotel's management can determine the reasons for rate fluctuations and how to influence them. Managers can include more factors into their analysis (such as weather data and dynamic competitors' prices) and make quick decisions on the fly.

To avoid disruption, top management should present the reasons for the shift to real-time analytics, clearly explain the opportunities it brings, and "infect" employees with these ideas. If there may be technical barriers, organize training so your employees feel confident in their ability to adjust.

Key takeaway: Even if you think that all changes are for the better, make sure your employees understand and share your knowledge about the background and benefits of the new system and are prepared to work with it successfully.

Final Thoughts

When you implement real-time analytics, you will face a variety of challenges. Don't be misled by the seeming simplicity of these challenges, and don't expect they will somehow solve themselves with time. The four challenges discussed here deserve your special attention; the key takeaways presented will help you address them and succeed.