TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

3 Data Management Rules to Live By

To make the most of our data investments, we need to go beyond simply managing data to deriving value from it. Here’s how.

• By Barr Moses
• January 25, 2023

Fans of The Office are no doubt familiar with head man Michael Scott’s theoretical book on business, which he cleverly titled “Somehow I’ll Manage.” Although he is at a loss for how to fill the chapters of this book (except for one, encouraging all of us to be at the ready with a stick of gum to share), Michael is crystal clear about the cover of his masterpiece: a photo of himself, with his sleeves rolled up, shrugging.

For Further Reading: Is Your Data Monetizable? 2023 Predictions: What Lies Ahead for Data Management and Analytics How the Right Data Management Foundation Fosters Digital Transformation Success

The word manage has taken on new meaning over the past few years, and it’s in line with the shrugging, disheveled imagery that defines Michael’s book cover. As a society, we’ve been “managing” ever since a worldwide pandemic (and, more recently, an economic downturn) has utterly changed our lives -- the way we work, shop, socialize, prioritize, and more. Just getting by the best we can.

Likewise, managing data in 2023 will look much different than managing data in the 1960s -- or even 2022. It’s no secret: today’s data teams are strapped for resources, often overwhelmed, and under pressure from executives to deliver value fast, particularly in this economy.

How can we overcome these hurdles and drive greater impact with our data -- in other words, go beyond idle data management to actual data value?

Here are three rules for organizations that want to win with data.

Rule #1: Position the data warehouse as a source of truth

What do you call a platform that is only accessible to one main group of users and holds data replicated somewhere else? A data silo, and in many cases, they’re the biggest threat to true operationalization.

Data silos can occur when:

• Business units put a premium on speed and autonomy over technical standards
• Data teams are sparse or overburdened, forcing business teams to work around existing systems or processes
• Data consumers purchase and implement their own point solutions rather than working directly with internal data teams

For example, you might have disparate databases such as a CRM for sales, CDP for marketing, and one for the finance team -- often with data overlapping -- with each business team slicing and dicing and manipulating data in those siloed environments. Not only do these data silos lock away organizational knowledge, expose compliance risks, and create waste from unnecessary duplicative architecture, they are often poorly run and optimized.

Instead, organizations must rely on their data warehouse, data lake, or data lakehouse as a direct representation and source of truth for what is occurring across the business. This way, data teams can ensure that reliable, accessible data is powering critical decisions and not just a hodgepodge of disparate and perhaps not-so-reliable sources.

Rule #2: Increase specialization and prioritize more roles in data

As companies continue to build out dedicated data teams and full-fledged data-centric organizations, look for a higher level of specialization to make its way to the management of the data stack. Here are just a few of the roles I expect to play a major part in managing the data stack in the future.

The data product manager is responsible for managing the life cycle of a given data product and is often responsible for managing cross-functional stakeholders, product road maps, and other strategic tasks.

The analytics engineer, a term made popular by dbt Labs, sits between a data engineer and analysts and is responsible for transforming and modeling the data such that stakeholders are empowered to trust and use that data. Analytics engineers are simultaneously specialists and generalists, often owning several tools in the stack and juggling many technical and less technical tasks.

The data reliability engineer is dedicated to building more resilient data stacks, primarily via data observability, testing, and other common approaches. Data reliability engineers often possess DevOps skills and experience that can be directly applied to their new roles.

A data designer works closely with analysts to tell stories about that data through business intelligence visualizations or other tools. Data designers are more common in larger organizations and often have product design backgrounds. Data designers should not be confused with database designers, an even more specialized role that actually models and structures data for storage and production.

Rule #3: Treat data as a product, but with some caveatsOne of the ideas with the most momentum in the data world these days is that of treating data as a product -- the notion that companies need to treat data with the same diligence and attention to quality as software teams use to develop modern applications.

This philosophy of treating data like a product makes sense, considering the ever-growing list of possible use cases (such as machine learning models) or even actual products (such as data-driven applications monetized by the business).

Best Practices

Here’s how data leaders can take data as a product from catchy PowerPoint fodder to actual real-world application.

Make documentation readily available. User-focused documentation detailing what the product is, how to use it, and why it works is a great first step. It forces you to answer the most basic question: what, exactly, is the product? It also helps shift the collective mindset from “this makes sense to me” to “will this be useful for my customer?”

Dedicate support and ownership to each data product. If we’re going to apply the same rigor to data products that we do to developing software applications, we should demand the same level of ownership and oversight of the products themselves. The data product model should be supported by dedicated engineers and analysts with specialized knowledge of the product and the business needs it’s addressing, just as applications have dedicated product managers who know the ins and outs, origins, and outlooks of each application.

Foster cooperation between data consumers and producers. Just as service-level agreements play a crucial role in setting expectations for application performance, it’s becoming a more common practice to set expectations for the performance of data products within a data contract. Data observability solutions that proactively measure and monitor data quality from end to end play an important role in helping data teams set, report, and meet the standards set in these data contracts.

Ensure internal customers are actually using it. Producing a data product without a customer to use it is like cooking a holiday meal for an empty dining room. Adoption is the true measure of your data product’s value. Be sure to track your monthly active users and find creative ways to evangelize within your organization.

With these best practices in mind, you can take treating data like a product from buzz phrase to an approach that actually helps deliver faster value with data.