What is data governance?How to do data governance?The author introduces and analyzes the advanced experience in the industry and personal practical experience.
With the explosive growth of enterprise data, enterprises will have problems such as data islands, non-standard data construction, inconsistent indicator caliber, unstable data query and calculation, and data security when using data. Therefore, various companies have also started data governance to promote the construction and use of data standards.
Next, the Straw Hat Boy will start a series of articles on data governance, combining advanced data governance experience and personal work practice experience in the industry to help everyone systematically understand data governance.
1. What is data governance?
As defined by the Association for Data Management (DAMA), data governance is the collection of activities (planning, monitoring, and execution) that manage data assets in the form of rights, rights, and control.
According to IBM's definition, data governance is a set of institutional and management activities to improve the availability, quality, and security of an organization's data through different policies and standards, and the goal of data governance is to maintain high-quality data that is secure and easily accessible to gain deeper business insights.
Explanation of terms:
- Metadata: Data that describes data, such as the storage location, model definition, and kinship of the recorded data, similar to a portrait
- Master data: Data that describes the core business entities of an enterprise, such as customers, products, accounts, etc., and has high business value and can be reused across various business units within the enterprise
The key points of data governance are data stability, standardization, and security, just like building a real estate, it is necessary to design the real estate architecture, lay a good foundation, formulate construction specifications, and promote the construction of each team member.
2. Why do we need to do data governance?
1. From the stage of development, look at the reasons for the development of data governance
(1) Phase I: 2005-2009
Around 2005, the early days of data warehousing emerged in China, initially dominated by commercial banks and telecom operators, and then joined by companies in the energy and other industries. Through cooperation with foreign IT consulting companies such as Accenture and IBM, commercial banks were the first to put the concept of data governance into practice in China.
Data warehouse construction involves extracting data from different platforms and integrating them, and in this process, it is necessary to ensure data quality, including data caliber, data standards, and data model unification. Through data governance, the establishment of data standards, data models and other management systems can improve data quality, ensure the smooth progress of data warehouse construction, and then better support BI and other data analysis applications.
Straw Hat Boy: The development stage of data governance is closely related to the development of big data, and as long as it involves the construction of big data, there will inevitably be problems such as data standardization, data quality, and data security.
(2) The second stage: 2010-2014
The data governance needs at this stage are mainly concentrated in the banking industry and are mainly driven by regulatory policies. In 2011, in order to promote the participation of Bank of China in the international clearing system, China promulgated the "Good Management Standard for the Quality Management of Banking Supervision and Statistical Data", which sets out requirements for bank data governance in terms of organization and personnel, system construction, system guarantee and data standards, data quality monitoring, inspection and evaluation, application and storage.
Straw Hat Kid: Financial institutions such as banks have higher requirements for the quality of underlying data and are facing certain regulatory pressures, which inevitably require data governance
(3) The third stage: 2015-2018
Around 2015, enterprises started to build big data platforms, and by 2018, the concept of data middle platform became popular, and the data middle platform included unified asset management, unified data element management and other content related to data governance. At this stage, more and more enterprises are starting to build full-time teams for data governance.
In 2018, the China Banking and Insurance Regulatory Commission (CBIRC) issued the Guidelines for Data Governance of Banking Financial Institutions, which includes data governance architecture, data management, data quality control, and data value realization.
(4) The fourth stage: 2019-present
Since 2019, the digital transformation of enterprises has entered the fast lane, and at this stage, data governance has been internalized as part of the construction of enterprise mechanisms.
For example, in September 2020, the State-owned Assets Supervision and Administration Commission (SASAC) issued the Notice on Accelerating the Digital Transformation of State-owned Enterprises, which clearly put forward the requirements for building a data governance system for the digital transformation of central enterprises. It includes clarifying the centralized management department for data governance, strengthening data standardization, metadata and master data management, and regularly assessing the maturity of data governance capabilities. Strengthen the dynamic data collection of production sites and service processes, and establish a data collection, transmission and aggregation system covering the entire business chain.
Straw Hat Kid: At the national level, state-owned enterprises will gradually build data platforms and carry out data governance, and the demand for data talents will continue to increase.
In industries with relatively mature data governance, including finance, communications, energy, manufacturing, etc., more enterprises have set up full-time departments and positions for data governance, and the more mature the data governance enterprises, the closer the full-time department is to the business side, and the higher the level of the full-time department.
In 2021, the Ministry of Industry and Information Technology (MIIT) released the 14th Five-Year Plan for the Development of Big Data Industry, which is guided by the release of the value of data elements and strengthens high-quality data governance.
2. The value of data governance to enterprises from the perspective of the current situation
Let's take a look at the phased problems encountered by the DataWorks team in the promotion and application of data construction.
(1) Initial stage: the contradiction between data volume and stability
With the growth of data volume, data stability is insufficient, task scheduling often hangs down with the increase of scale, and cluster computing resources are insufficient, resulting in a long processing time.
(2) Application stage: the contradiction between data inclusion and use efficiency
In the application of data, the number of people using data increases, the data warehouse personnel are tired of obtaining data, and the pressure on the data team increases, and the construction of the data warehouse will gradually become chaotic, which will hinder the efficiency of subsequent data use.
(3) Scale stage: flexible and convenient and risk control
There are many data exports, it is difficult to control human leakage, security needs to be improved, and the management actions of various data security are often contrary to efficiency, and it is difficult to classify and grade data.
(4) Mature stage: the contradiction between business change and cost governance
The pressure on data cost is high, and I don't know where the data cost is, so I dare not delete the data and dare not set the task.
Straw Hat Kid: State-owned enterprises are gradually promoting digital transformation and data governance under the guidance of policies, while private enterprises are gradually promoting data governance driven by the growing growth of business data.
3. How to do data governance for different types of enterprises
Data governance in traditional enterprises is closely related to digital transformation, and data governance actions are often integrated into every aspect of data construction.
(1) Internet
Taking Alibaba as an example, Alibaba has built the DataWorks big data development and governance platform, which has built capabilities such as standard design of indicator data warehouse, data analysis, metadata management, and data security management.
(2) Energy
Including coal, electricity, photovoltaic, power stations, etc., such as State Grid, Sinopec, etc.
The data governance of the energy industry, such as electric power, cannot be based solely on traditional IT technologies such as data quality, lineage analysis, and metadata management, but needs to be closely integrated with business and closely coordinated with all business links in data governance.
For example: equipment maintenance: the power grid company carries out research on predictive maintenance through massive data mining of equipment operation, predicts the future state of equipment operation with high accuracy, and predicts the possibility of equipment failure, so as to achieve the purpose of guiding maintenance based on equipment status.
Common challenges, including:
- Multiple data types: IoT devices widely deployed in the power industry generate multi-source heterogeneous data with different formats, frequencies, and definitions
- Massive data, strong timeliness: The generation, transmission and consumption of electricity are changing rapidly, and massive data resources can be generated in a moment.
- Long data link: There are many participants in the integration of the energy value chain, and there is a close connection between value activities.
Key governance methods, including:
- Automatic collection of various types of data asset information: Different collection adapters are developed according to different data sources to achieve automatic collection of different types of data
- Comprehensive management of enterprise data assets: After the comprehensive collection of enterprise metadata, technical metadata, and model metadata, enterprises should do a good job in sorting out the overall data assets and managing data quality
- Data asset servitization: Provide self-service query service and automatic data acquisition service
(3) Finance
Financial data governance is an important entry point for the digital transformation of the financial industry, and it is also a key point to promote the transformation of the financial industry from high-speed growth to high-quality development. The Guidelines for Data Governance of Banks and Financial Institutions issued by the China Banking and Insurance Regulatory Commission (CBIRC) are based on data governance teams, data management standards, data quality control, and data value realization.
Common challenges in financial data governance include:
- Contradiction between data value discovery and data information protection: The Data Security Law and the Personal Information Protection Law have been implemented one after another, requiring financial institutions to integrate the security and privacy protection of individual customer information into the entire process of data collection and application.
- Financial institutions, such as banking institutions, have problems such as many branches in various places, fragmentation of operating data, and difficulties in data circulation
Key Governance Methods:
- Governance servitization: The management concept is transformed into a service concept, and the business is provided with comprehensive services by providing convenient tools or services.
- Governance process: Combined with the development process, the data model and data dictionary management are streamlined, and data standards are implemented from the data source.
- Scenario-based governance: Combined with regulatory reporting, the quality of reporting is monitored throughout the process.
- Open standards: Build a data standard operation system, explore more open and shared scenarios, and apply data governance achievements.
- Asset intelligence: Build intelligent data asset management and combine advanced technologies to fully release the value of data.
4. Opportunities for data talent
Under the trend of the state promoting the digital transformation of enterprises, not only Internet companies need data talents, but also many traditional enterprises, including energy, communications, finance, manufacturing, etc.
Enterprises face different problems with different degrees of informatization and data, and the ability focus of the required data talents will also be different. However, in the overall digitalization promotion, certain general capabilities will also be required, such as data collection, data analysis, data computing, data governance, and data application.
Therefore, in terms of work practice, individuals can continue to increase their practice and understanding of the big data platform and each module, and grasp the entire big data system as much as possible.
Columnist
Straw hat boy, public account: a data person's own place, everyone is a product manager columnist. Author of the book "The Road to Big Data Practice: Data Middle Platform + Data Analysis + Product Application", focusing on the field of user portraits.
This article was originally published on Everyone is a Product Manager. Reproduction without permission is prohibited.
The title image is from Unsplash and is licensed under CC0.
The views in this article only represent the author's own, everyone is a product manager, and the platform only provides information storage space services.