TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Seven High-Priority Areas for Data Integration Modernization

Like many IT systems, data integration infrastructure needs to modernize to keep up with new business and technology requirements.

• By Philip Russom
• February 17, 2016

All of us in data management are experiencing an extended period of great change as big data, other categories of new data, and new data management platforms enter our organizations. In response, most user organizations are scrambling to learn the new technologies and -- more important -- how to leverage the new data and platforms for business advantage. As a result, many data professionals are now facing new requirements as well as future requirements that will arise as new data sources come online.

The changes afoot are driving many technical organizations to rethink and modernize their data management infrastructure, team, and skills. Among these efforts, data integration modernization is one of the most pressing because of the broad role data integration (DI) plays in capturing, processing, and moving data -- both old and new, for both analytic and operational purposes.

Without modern DI solutions, organizations cannot satisfy new and future requirements for big data, analytics, and real-time operation. DI modernization can take many forms depending on the current state of your infrastructure and what kinds of new data or platforms you must embrace. Instead of trying to list them all, here are seven areas where users commonly or urgently need to modernize their DI. This list can help you prioritize your modernization efforts as you select vendor products and update your solution designs.

Multiple data ingestion techniques allow data to move at its own speed or generation frequency. That way, data arrives in target data platforms as soon as possible and is available for immediate business use in dashboards, reports, and analytics.

Data prep enables a data analyst, data scientist, or similar user to construct a data set prototype quickly without being slowed down by excessive modeling and standardization. Such speed is critical to modern analytics practices.

Self-service data access helps users work with spontaneity and speed because they aren't waiting for IT or a data management team to construct a data set for them. This is key to such modern practices as agile development, data exploration, and data discovery.

New data platform types, when incorporated into a modern data integration infrastructure, provide new options for capturing non-traditional data and massive volumes of data, as well as for analytic processing and DI transformations. This includes new platforms based on event processing or Hadoop, plus new persistence practices such as data lakes, data vaults, or enterprise data hubs.

Right-time data movement is the secret sauce that accelerates many time-sensitive business practices, including operational BI, performance management, and a wide range of real-time analytics. Because there are many "right" times for moving data, proper enablement typically involves multiple data integration functions that operate at multiple speeds and frequencies.

Non-traditional data promises great business value for decision making and analytics. To support that goal, a modern data integration platform must capture data pushed to it, handle unstructured data, support new approaches to metadata, and coordinate with tools for natural language processing.

Integrated tool platforms include many tool types for data integration, data quality, master data management, and event processing. The tools are tightly integrated to facilitate collaboration among developers and to foster the design of modern DI solutions that call multiple, highly diverse tool functions.

To learn more about data integration modernization, read TDWI's recent Checklist Report, Modernizing Data Integration to Accommodate New Big Data and New Business Requirements, online here.