One
background
Quality assurance is a systematic engineering based on software testing, which follows the law of gradual development. By formulating the landing strategy according to local conditions, designing the scenario plan, obtaining the test results, and repeating the cycle. In the end, with the joint efforts of every DEWU test engineer, a set of quality assurance methodology adapted to DEWU technology has been accumulated, that is, the DEWU quality management system introduced in this article.
Two
Build a "four modernizations" system
After three years of construction, the quality management system now has a complete mechanism, process, method and tools. The goal is to achieve standardization, online, automation and intelligence, and finally form a three-dimensional quality management system. Thereinto:
- "Mechanism" refers to the (R&D) quality assurance mechanism. Based on the "iterative quality review mechanism", each iteration is carried out and the online risk is controllable before it can be approved.
- "Process" means: testing & collaboration process. It is divided into a testing process and a collaborative process. the application of testing processes to reflect the quality of R&D in an unbiased manner; Supplemented by test quality calibration. The effective application of collaborative processes will help improve the quality of testing. Therefore, collaborative quality is introduced to stabilize the test quality and ensure the stability of the final line.
- The "method" is the implementation method. With the "Quality Month" activity in the business domain as the main carrier, we continue to optimize the quality of the R&D process, including cross-domain review, CR merger, on-time testing, smoke passing, defect daily clearance, defect introduction, defect escape and other basic indicators.
- The "tool" is: the quality engineering tool. Focusing on stability, quality, efficiency, safety, experience, and compliance, we will improve tool support capabilities. The end result of each tool is to the core, such as: RDC is used to standardize, online, and partially automate the R&D process to improve collaboration efficiency and optimize delivery "efficiency"; As an observability tool for R&D process quality insights, the quality dashboard serves the process "quality". The existing tools (R&D toolchains) are shown in the following diagram:
- The "iterative quality review mechanism" is a unified review mechanism for the quality platform. Relying on the quality dashboard as the main tool, combined with the quality assurance portfolio strategy of each business domain, we focus on the quality and efficiency drill-down analysis of each iteration version. Identify the risks that affect delivery in the current iteration and the corresponding mitigation measures to ensure that the release is carried out under the condition of low risk or controllable risk.
- "Built-in quality" is commonly known as "R&D quality", and it is officially solidified after testing. "Test quality" reflects the comprehensiveness of the test coverage through the level of defect escape, and at the same time, calibrates the quality of the R&D. "Collaborative Quality" is used to stabilize the quality of the test and ensure the final stability online. Here are some examples:
- The test execution schedule risk caused by the failure of the demand to be tested on time;
- Frequent process changes can lead to estimation deviations, rework losses, and even missed change overrides.
- The "Quality Month" special activity is fully promoted in all business domains of Dewu Technology, and through the "pre-event" application of the cross-domain use case review mechanism, the upstream and downstream collaborative technology implementation and verification coverage scenarios are connected; The "iterative quality review mechanism" is applied "in the process" to effectively assess the quality risk of the process and strictly control the quality of the required requirements; The "online event review mechanism" is applied "after the fact" to form a result verification to guide the detailed optimization and improvement of the next iteration in the development, testing, release, and production stages. After achieving comprehensive results, the keyword of "quality" advocated by the activity has long been integrated into the consciousness of every technical person, and the high-quality delivery standard is practiced on a daily basis.
- The relationship between "mechanism, process, method, tool" is defined as the mechanism that sets the overall framework and objectives of quality management, the process defines the specific steps to achieve these goals, and the method is the specific implementation technology adopted in each step of the process, and the tool is the auxiliary equipment or software required to implement the method and process. These four concepts are interdependent and mutually reinforcing, and together form the overall system of R&D quality management, ensuring that every aspect of products and services, from design to delivery, meets predetermined quality standards.
Three
Objectives and framework of the mechanism
Quality is the foundation of the quality assurance mechanism, and stability is the natural result of effective quality activities. Through reasonable fault definition and classification as the traction goal, the principle of no major problems (failures), rapid recovery of minor problems (smoke points/events), and the impact of failures is minimized as much as possible (fault impact = fault impact area x fault repair time).
With the rapid development of the business scale, Dewu App availability, asset loss prevention and control, privacy compliance, data security and other aspects are facing severe challenges. At the same time, according to the level of production failures, the technology is divided and conquered according to the characteristics of the business, and at the same time, the production failure level is gradually downgraded to achieve the effect of zeroing, and the total number of production failures has been continuously reduced. As shown in the figure below:
The framework of the quality assurance mechanism is described as follows: the quality objectives of the quality management white paper V2.0 are clarified, and the corresponding implementation specifications and supporting requirements are concretely defined. In the iterative process, the iterative quality review is applied, the risk is identified before the release is approved, and the drill-down attribution diagnosis is made, and specific measures for risk degradation or risk avoidance are formed, and implemented before the launch; Finally, through iterative quality review, the typical scene of the process is restored, and inferences are drawn from one another; At the same time, the adaptation of the target and its supporting specifications is calibrated with the results of online operation. The whole process is shown in the following figure:
Then, by repeating the above cycle, a spiral upward effect is formed, and the rhythm progresses to a higher goal.
It is worth pointing out that after hundreds of iterations of optimization, the quality of R&D has been significantly improved by more than 60%, and the technical debt has been greatly reduced by 70%; The basic quality indicators of the process, such as the on-time test rate and the smoke pass rate, are stable at more than 99%.
Four
Processes & Specifications
Dewu project management, through sponsor to distinguish two types of business and technical requirements; At the same time, it is delivered in iterations or projects according to the complexity and scale of requirements. For four combinations, namely Business Iteration, Business Project, Technology Iteration, and Technology Project. The testing process and specifications are slightly different, and the quality assurance process and specification for business/technology iteration requirements and the quality assurance process and specification for business/technology independent projects (including small projects) are formulated respectively. The following diagram shows the general test process and specifications:
Among them, the quality assurance plan for the business/technical transformation needs of the quality platform is specially formulated, the key test actions, approval standards, and tool platforms are defined, and the test team is guided to carry out the work in detail, as shown in the following figure:
In the process, based on case-driven optimization and improvement, the implementation plan V2.0 of the cross-domain use case review process and the cross-domain use case review mechanism of the standardized quality platform was precipitated to avoid the uncovered boundary problems caused by horizontal collaboration between multiple business domains.
In addition, for business/technical projects, formulate a quality assurance plan for the business/technical transformation project of the quality platform, and refine the responsibilities and implementation specifications of the test owner. For the scenario of business brushing, formulate a business brushing test plan, clarify the process specifications, and reduce the change risk caused by high-frequency business brushing changes.
Five
Methods & Tools
Test cases are the most basic test assets, just as code is to APIs, and APIs are to distributed systems. With the help of various tool platforms, it can be transformed or processed into various forms of "test products". The application of various "test products" relies on the research and development of the business domain of Dewu and the landing practice of test students, and through three years of continuous accumulation, a hundred schools of thought have formed a situation of contention and vigorous development. Here are some typical practices:
- Interface automated testing, as the main method of regression testing, is also a classic test execution efficiency improvement method. Interface automation platform for development self-testing, smoke (admission) and integration regression phases. The ROI presents a linear relationship, and for the high-frequency iteration mode, automated testing can ensure the availability of the core link. In particular, it maximizes the benefits of technical transformation projects such as service/interface migration and architecture governance, and escorts the repayment of technical debts.
- Traffic playback, as a pre-launch testing method, is applied to the pre-release stage through the traffic recording and playback platform. Through simple production recording and pre-transmission playback, a more realistic user scenario coverage can be obtained at a lower cost. Combined with the interface coverage, the incremental traffic is selected or constructed to the traffic pool to make the distribution more comprehensive. Due to its low-cost application method, it can be used for pre-launch testing at a low cost; In addition, changes such as service/data migration and local refactoring are extremely application-friendly, greatly reducing the test investment.
- Accurate testing, as an auxiliary test analysis and execution method, can improve test coverage and measure more accurately. Through the accurate test platform, it is applied to the smoke stage (after the test), according to the code change, with the maintained label and binding relationship, such as the asset loss field, the asset loss code, (overstepping) sensitivity, etc. Test case execution coverage is recommended. If there is a difference, i.e., an interface that is not associated, a CR is often required and the coverage plan is evaluated. Its application can not only verify the accuracy of the change scope evaluation, but also evaluate the comprehensiveness of the test analysis, and form a positive feedback for the understanding of the business/system module.
- Monitoring alarms are used as a method to assist offline test execution, check and fill gaps, intercept abnormal defects, and measure more carefully. Through the monitoring platform, it is applied to the offline test execution stage. Develop and test students to jointly formulate business alarm rules to achieve alarm aggregation and convergence. On this basis, alarm insight and defect transformation are realized, and they are benignly applied in the T or T+1 daily clearing method.
- Front-end inspection, as an active response method for user experience, pays attention to the user's feedback on the user side during daily inspection, and discovers and repairs it early. Through the front-end inspection platform, the stock URLs are imported through the TOP access records of the last xx days, and the scheduled task inspection is set. Increments are imported through precise URL recommendations, and inspections are covered. JS Error exceptions can be intercepted in advance in the pre-release phase, and early detection and intervention can be achieved in the production and operation phase, and the JS Error can be intercepted before the user.
- Dcheck, as an effective method for asset loss interception, real-time reconciliation, early detection and early stop of bleeding. Through the Dcheck platform, it is applied in a combination of online and offline. Offline drills of asset loss scenarios, and can play a role in preserving the freshness of script rules; Real-time online monitoring and abnormal alarms.
In terms of tool application, it is worth noting that there are three costs that need to be continuously optimized to be gradually reduced:
- Cost of "conversion" of test cases (development effort)
- Cost of "supporting" each other between tools (integration effort)
- The cost of "applied" each type of tool (ease of use)
The following figure shows some efficiency optimization points and optimization ideas.
Finally, with the effect of clear atomic capabilities and convenient combination and application of tools, test engineers can gradually focus on in-depth understanding of business systems and detailed review of technical implementation, so as to complete reasonable test strategy formulation and agile implementation.
Six
Focus on "keywords"
As the focus of the official plan, the "keyword" of Dewu technology is the basis for the value judgment of the complex work, the central idea of the stage review and summary, and the outpost and lighthouse of effective business support, guiding the technical work to do it right and well. Start with the end in mind, practice unremittingly around the "keyword", and continue to harvest.
- Stability, in general, is a natural consequence of effective quality assurance. It is often led by the stability or O&M team to conduct real-time monitoring and emergency response to faults. The number of production failures and their distribution are the main approval indicators. Policy compliance: No major problems, small problems are quickly recovered, and the impact of the fault is minimized as much as possible, that is: the impact of the fault = the surface affected by the fault x the time of fault repair.
- Efficiency, through the business delivery capability, plan support capability and process collaboration capability from the bottom up to achieve the purpose of efficiency, that is, to present "more" and "fast" with data, combined with NPS to fit the business experience.
- Experience, improve the experience of using Dewu products for users. It makes the App better to shop, the goods to be bought/sold better, and the after-sales service to be more assured. For the Dewu App, the startup, first response, performance, package volume, security and privacy compliance and other aspects are measured, the baseline anti-degradation and the rating of competing products are tracked.
- Growth, software engineering is a practice-oriented discipline, practice, continuous practice, and the accumulation obtained in practice is the acquisition of engineer growth. According to the Feynman learning method, think clearly, write out, and explain clearly through sharing in combination with blogs.
- Innovation: Through "combinatorial innovation" and "AI innovation", we can dig deep to find effective scenarios in terms of stability and efficiency. In terms of stability, the focus is on coverage comprehensiveness and coverage granularity; In terms of efficiency, the focus is on implementation and operational reduction.
Seven
summary
- The mechanism sets the overall framework and objectives of quality management, the process defines the specific steps to achieve these goals through the specific implementation techniques adopted at each step of the process, and the tools are the auxiliary equipment or software required to implement the methods and processes. These four concepts are interdependent and mutually reinforcing, and together form the overall system of R&D quality management, ensuring that every aspect of products and services, from design to delivery, meets predetermined quality standards.
- The framework of the quality assurance mechanism is described as: the quality objectives of the quality management white paper are clarified, and the corresponding implementation specifications and requirements are concretely defined; In the iterative process, the iterative quality review is applied, the risk is identified before the release is approved, and the drill-down attribution diagnosis is made, and specific measures for risk degradation or risk avoidance are formed, and implemented before the launch; Finally, through iterative quality review, the typical scene of the process is restored, and inferences are drawn from one another; At the same time, the adaptation of the target and its supporting specifications is calibrated with the results of online operation.
- Test cases are the most basic test assets, just as code is to APIs, and APIs are to distributed systems. With the help of various tool platforms, it can be transformed or processed into various forms of "test products".
- With the clear atomic capabilities of the tools and the convenient combination and application of the tools, test engineers can gradually focus on the in-depth understanding of business systems and the detailed review of technical implementation, so as to complete the formulation of reasonable test strategies and agile implementation.
- As the focus of the official plan, the "keyword" of Dewu technology is the basis for the value judgment of the complex work, the central idea of the stage review and summary, and the outpost and lighthouse of effective business support, guiding the technical work to do it right and well. Start with the end in mind, practice unremittingly around the "keyword", and continue to harvest.
Author: Bruce
Source-WeChat public account: Dewu Technology
Source: https://mp.weixin.qq.com/s/OxclNIXYXV--uozr1z3QWw