The original article "Big Data Platform Technical Architecture Planning Scheme" PPT format, mainly from the big data processing technology, big data target architecture, construction ideas, big data platform architecture, construction priorities, etc. Suitable for pre-sales project report and leadership report.
Material collation or source network public channels, if there is infringement, contact for quick deletion, more refer to the public number: Youxiang Think Tank
First, big data processing technology
Big data poses challenges to traditional data processing technology systems
Under the traditional analysis system architecture (RDBMS + minicomputer + high-end array mode), traditional databases cannot support massive data (such as more than 100TB, performance degradation), unstructured data, and the existing IOE architecture cannot scale linearly and is costly.
Second, the big data target architecture
In the current situation of a wide variety of data and complex data processing, it is not suitable to use a single technology to solve all problems, and the big data platform needs to use Hadoop resource pool, MPP database, and stream processing resource pool to mix and match big data technology architecture.
The data platform is based on cloud computing and big data technologies such as MPP, Hadoop, and stream processing
• DW database for analytical processing statistical analysis OLAP applications
• MPP database for correlation analysis of structured data.
•Hadoop platform software is deployed in Hadoop big data processing clusters, realizing massive unstructured data storage and processing and vertical aggregation of structured data.
•The flow data and complex event processing (CEP) rule engine platform is used to process data streams in real time, realize access and real-time processing of high-speed data streams, and detect key events in real time
Third, the idea of construction
From easy to difficult, steady progress: In the early stage, data integration is the mainstay, and data services are gradually provided internally and externally.
Control architecture, synchronous promotion: Synchronous promotion of data standardization and organizational reform, laying the foundation for the commercial use of big data sharing platforms.
Independent control and internalization: Gradually cultivate self-research teams and build integrated R&D and operation capabilities.
Fourth, the architecture of the big data platform
The technical architecture of the enterprise-level provincial big data platform includes four layers: data collection, data storage and computing layer, development framework and application center, and unified operation and maintenance management to provide services for various users. In the big data technology architecture, the storage and calculation of data are closely linked.
5. Key points of construction
Building focus 1 - relationships with other analytical platforms
Big Data Sharing Platform:
Network-wide XDR data acquisition, standardization, and full storage (1 month)
Network-wide network management data collection, standardization, and full storage
The big data sharing platform is responsible for the unified centralized collection and preprocessing of xDR data and network management data; Provides query responses from upper-layer applications to xDR fine-grained data.
Multi-dimensional small-grained aggregation, data integration, and storage according to application requirements
Provides detailed data query and light summary data query.
Performance Management System:
Obtain the full-hourly summary data required by the application from the big data sharing platform.
Data caching layer: responsible for in-depth processing and caching of data from big data sharing platforms; Provides a variety of aggregated data storage, processing and sharing, as well as comprehensive analysis and deep mining for the application layer.
Application layer: Carries various upper-layer application software and third-party applications to implement upper-layer applications.
Construction focus 2 - Develop data governance rules
• Collect directly from the data source according to the rules to avoid repeated data collection.
• Exploit the residual value of silent data for data already collected by existing systems.
• For data not collected by existing systems, increase collection points and explore the value of data.
• The data storage after collection follows the principle of localized storage of each domain, and the data warehouse of each domain is a public warehouse and is shared and used by the whole company.
• Unified data cleaning according to the rules, data distribution and authority control according to the needs of different professional applications after cleaning.
• If the missing data cannot meet the application requirements, either modify the data cleaning rules or re-collect the data.
• Explore the application scenarios and unknown values of big data internally and externally from the shared data and label combinations of various domains.
Construction focus 3 - HADOOP server measurement model
Model building: According to the HDFS storage capacity calculation, it is mainly divided into two aspects:
First, when a certain volume of data adopts different data processing technologies, it requires the physical storage capacity, that is, the theoretical calculation of disk raw capacity;
Second, for a certain configuration, the x86 server can provide effective storage capacity when it carries different data processing technology entities.
Finally, the number of X86 servers that need to be configured for a certain amount of data using different data processing technologies = physical storage capacity ÷ effective storage capacity that X86 servers can provide.
Construction focus 4 - HADOOP cluster 1/2 choice of site site