Solving 5 Big Data Governance Challenges in the Enterprise

“Good” has never been the same as “great.” Studies show a “good” understanding of enterprise big data has the same average odds as a coin toss. Understanding data is the first step for the enterprise, but it’s not the final goal. Organizations need data governance, a framework for decisions on data and related subjects. 

Fourteen percent of organizations feel they have a very good understanding of enterprise data, per our 2019 data quality survey. Over one-third define “good” as understanding 50-70 percent of data. Understanding 50 percent of data assets is a coin toss, not a foundation for analytics. 

Data governance challenges: traditional frameworks can’t scale

Formal governance is a concept that can offer significant benefits to the enterprise. It’s also required by the CCPA and other legislation. Traditional frameworks for data governance work on smaller volumes of structured data. Enterprises can face many challenges trying to govern the big data ecosystem. Here are five to consider.

1. Data governance isn’t simple

Governing the quality of structured data is easy, especially compared to social media or sensor data. Unstructured data is messy and it moves fast. 

 Unstructured data contains many quality dimensions. Governance should account for every point of potential failure in data assets. This includes risks like sensor signal loss, noise, and incorrect data defaults. 

2. Context is mandatory

You can’t understand the impact or risk of data assets without context. Compliance, security, and business value are three examples of critical business context. Each can impact data integrity, storage, and access requirements, like masking Personally identifiable information (PII). 

3. Consistent enforcement is challenging

In today’s dynamic, multi-cloud big data environments, organizations need automated policy enforcement. Monitoring data and metrics is the only way to understand changes to compliance.

4. Everything is relative, especially data metrics 

Quality is relative to the business context. For example, social media open-source intelligence (OSINT) used for directional market analysis isn’t a critical asset. It doesn’t contain PII, and it may have relatively few governance requirements.  In contrast, training data for a hiring algorithm could be subject to stringent quality and privacy metrics.   

5. Siloed data governance efforts are bound to struggle

Governance efforts have little impact when they’re walled off from technology, people, and process.  

Cross-functional collaboration is the only way to create an effective framework and metrics. Even organization-wide transparency may not be enough. Governance can’t exist separately from data quality monitoring technologies.  

You can’t govern big data without quality and profiling

Governance translates business strategy and regulations into policies and metrics. Collibra creates an interactive catalog of data sources and business definitions to spark cross-functional collaboration between data stewards.  

Trillium Quality can profile big data environments at scale, providing a steady stream of insight for Collibra. Trillium can translate business rules into applied logic. It can inform better governance by continuously measuring metrics, policy, and risk.

An effective big data governance framework requires real-time profiling, quality, and business context. A symbiotic relationship between governance and quality technologies is the key to success. 

To learn more, read our eBook: Fueling Enterprise Data Governance with Data Quality