For many individual users, PC products are mature, unchanged, and redundant in performance. But for commercial PCs that account for more than half of the market share, whether it is the industry habits of regular updates, or the strong market demand in the past two years, coupled with the "just in time" hardware and application revolution, the commercial computer market can be described as unprecedented prosperity
Compared with the diversification of consumer computer user needs and application scenarios, commercial computers reflect more of the specialization of applications. The former has also formed a variety of performance evaluation methods, corresponding to insufficient representation, limited degree of specialization and low data credibility, etc., taking the game as an example, the number of games has a large number of performance characteristics, and the performance is affected by driven optimization, parameter settings, the test results of a single game are difficult to be representative, and the type of representativeness is not available. In addition to games, software with performance testing functions: there are many kinds of benchmarks and various marks, which either advertise targeted testing of a function or simulate an application scenario/process.
Today, we are going to introduce commercial PC test software for future PC application scenarios. Commercial PC test software has different sentence reading interpretations: commercial PC test software and commercial PC test software.
The former is relatively more diverse, individual consumers often use PCMark, CrossMark, GeekBench, XPRT series and other products for the entire PC system, there are also 3DMark, Sandra, Cinebench, ADIA, CPU-Z, Lu Master, Antutu, etc. for a certain feature or component test, and even many games provide built-in benchmark functions, which can also be counted as test software to a certain extent. With the formalization of "testing", the development scale of test software is getting larger and larger, and some related development companies have also shifted from the initial shared, open source methods and studios to formalized and corporatized development.
First of all, the development process of commercial test software must be commercialized rather than workshop-style, requiring extensive participation and cooperative development of industry-related enterprises, each contributing their own leading products and technologies, related optimization and parameter settings, which ensures that the test software is fair and representative, and the software product life cycle is long and the conclusion stability is better. At the same time, the data of the relevant test, including but not limited to the system platform, parameter configuration, and test conclusions, are verifiable, verifiable and tamper-proof.
Testing the software of a commercial PC, in addition to being commercial, the more important factor is that the test scheme it uses is commercial-grade, and the simulation run program is developed by isvV and has versatility. At the same time, the test method/script is public, that is, no automatic process/script is used, only the same software is applied, and the same evaluation conclusion can be obtained. In addition, for commercial PC usage scenarios, the test mode or parameters of related software can be adjusted and set through GUI, command line or code to meet the industrial test requirements such as batch and unattended. Qualified commercial PC test software, like the commercial PC itself, has higher requirements for the stability and reliability of test conclusions, and the data deviation of different test cycles is smaller under the condition of unchanged parameters, and the general requirement is less than 1%.
Products that can be called commercial test software for commercial PCs are very rare in comparison, and there are considerable thresholds for using themselves and data analysis. How "high" the specific threshold is, we will use the next article to introduce, today's protagonist is the commercial PC test software itself.
The leader of commercial PC test suite products is a small company called BAPCo (Business Applications Performance Corporation), and the fist products are also various "Mark". Specifically, there is mobileMark for simulated laptop (including battery life testing), SYSMark for focusing on the performance of office application processes, and its simplified version of CrossMark. Since the release of the last version of SYSMork2018/MobileMark2018 in 2018, Windows 10 has been a mainstream PC platform, and with the official release of Windows 11, BAPCo has also officially released the "25" series of products, which is enough to see the large change in its characteristics from this number span, which is concentrated in the test application load and new hardware support (such as instruction scheduling mechanisms), such as the same laptop, In MobileMark 2014, the battery life can break through, 24 hours, and the 2018 and 25 versions, step by step.
The name of BAPCo has clearly pointed to the field of commercial applications, and the biggest feature of its products is based on genuine, mainstream commercial software such as Microsoft, Adobe, and Google, just like UL Procyon. SYSMark25's workflow is more complex, with popular tools such as Audacity, AutoIt, and WinZip for Enterprise, which are a good reflection of the performance status of PCs in office environments.
Compared with the thunderous PCMark, SYSMark / MobileMark can be described as very low-key, its birth is to serve commercial testing, so ordinary consumers know very little about them, and their test conclusions, often appear in the PC parameter table "*", is an important parameter to identify the basic performance of the product. BAPCo products have a higher test cost, in addition to no piracy, thousands of dollars without discount prices, blocking a lot of "play" test players, the early version only supports the English system, script simulation workflow is prone to failure (taking into account compatibility testing), often 2 hours of installation + 8 hours of testing time, which is not the reason for being abandoned by individual users!
BAPCo is aware of these issues in the cost of testing use, and SYSMark 25/MobileMark 25 promises to support Chinese Simplified systems through upgrades in the future. PC products sold in China are generally pre-installed with a low price of single-Chinese Simplified Windows Home Edition, imagine, if you want to meet the English test environment, you need to upgrade to the professional version, switch to the English system, remove the Chinese language pack and related features, uninstall the unrelated functional components of Windows as required... This set of operations takes up to 2 hours to complete, and there is a higher risk of test failure. The DIY platform installs its own English system, and integrates a large number of OEM-optimized whole machines, especially laptops, so the operation is very unprofessional.
The way BAPCo can really solve the extreme cost of SYSMark testing is to launch CrossMark and go a step further. This new test software can be used across different operating systems and hardware platforms such as Windows, Andorid, mac OS/iOS and Linux, not only the regular version has a complete test process and performance evaluation system of the professional version, but also is provided to all users for free through integration in the official software store such as Windows Store/App Store/Google Play. CrossMark is the only test software on the market that can provide terminals and desktop terminals with different software and hardware platforms, and the performance evaluation and comparison of the whole machine. Using the same workflow and load, and comprehensively comparing productivity, creativity and responsiveness, CrossMark, which continues the SYSMark/MobileMark testing concept, has a great similarity with the former, achieving more than 80% evaluation accuracy at a 1/10 time cost. The so-called test accuracy is not the poor stability of the test data, on the contrary, under the same parameter setting, the crossMark score deviation of the same product is much less than 1%, which is a high level in the industry. The evaluation accuracy is related to the length of the simulation process and the complexity of the task simulation, which is inversely proportional to the time cost. With CrossMark, users can make preliminary, light-load evaluations of PC products, macro evaluations of product performance in different scenarios, and more comprehensive and high-intensity evaluations can be completed by SYSMark.
The CrossMark online database provides a rare ranking of the highest performance system performance of various mainstream processors on the market, and when expanded, you can also see the results of all the same processor platforms that submitted test results, and the main hardware parameters are listed. Only data that passes the data accuracy and correctness test will enter the ranking.
With low testing costs (cross-platform, multilingual, free, short testing time, and high results reliability), CrossMark gained the attention of many players as soon as it was launched. In just a few months, the test results in its online database have been very rich, and the mainstream architecture processors are readily available.
By adjusting the test parameters, the user can set all or one of the three test suites running CrossMark, or set the number of runs to achieve more targeted tests. In general, crossmark's creative test projects in audio, image, video, Flash and other multimedia material processing mode is closer to the workflow, the load is lighter than SYSMark's corresponding project, and it is also smaller than the Procyon photo editing and video editing module, which is more suitable for evaluating lightweight work.
In addition, CrossMark has also added AI processing features in a timely manner to facilitate the evaluation of future use scenarios.
Free is free, but only in the standard version - forced upload of test results, which also leads to frequent abnormal test results, fortunately, BAPCo has a Pending check mechanism, which can quickly remove abnormal data. For commercial test customers, the Pro version can be purchased, which prohibits automatic data upload.
CrossMark's strengths are also reflected in its relationship with SYSMark, which has a high threshold, while the former achieves 100% identical test items and data accuracy. Such a low-cost, high-quality CrossMark, the future can be expected!
The second thing to introduce is UL Procyon. In 2014, UL acquired Futuremark (MadOnion, Crazy Onion), which has made great achievements in the consumer PC market with 3DMark and PCMark series, and officially entered the benchmarking service market. As pcmark products continue to be used for industrial testing, the negative impact of data stability and analyzability during open source/shareware foundation and pipeline testing is amplified. Until the end of 2020, UL has successively launched a new commercial test software Procyon that meets the requirements of industrial assembly line testing, entering the field of commercial PCs and professional PC testing.
The UL Procyon benchmark kit includes 3 PC test components and 1 Android test component, the latter is an AI inference benchmark to evaluate the AI performance of mobile terminals, and it remains to be seen when it will land on the desktop platform.
UL Procyon's three components for PC products are: Office Productivity Benchmark, Photo Editing Benchmark, and Video Editing Benchmark (benchmarks.ul.com/zh-hans/procyon). As the name suggests, the usage scenarios they target are very clear and unambiguous. Photo editing benchmarks and video editing benchmarks are for DCC (Digital Content Creation Digital Content Creation) users, both support the latest version of related commercial software (testers themselves purchase and install), the former imitates the professional photographer and editing workflow, the digital negative (Digital Negative) into Adobe Lightroom Classic, for a variety of retouching, cropping and stretching, Then use Adobe Photoshop to edit and layer effects in the picture set. The video editing benchmark uses Adobe Premiere Pro to import and edit multiple videos, and the adjusted and inserted intermediate videos are also exported using H.264 and H.265. These two tests can be seen as a professional upgrade to the DCC module in PCMark.
The Office Productivity Benchmark is the core module of commercial PC testing, and it was officially released in October 2021. It uses 4 sets of words, Excel, PowerPoint and Outlook in Microsoft Office to simulate daily office applications, which can be said to be the daily life of thousands of office workers. Of course, you could argue that I still use this or that software when I go to work, such as video conferencing, browsers, chats, and even streaming software. These rely on real-time web application performance testing and are pending implementation of Procyon's new feature modules.
SPEC (Standard Performance Evaluation Corporation) is a 30-year-old veteran testing service company, its test software is numerous, the most famous of which are SPEC CPU, SPECapc suite, SPECviewperf suite and other products.
Evaluation (evaluation) is the core competitiveness of SPEC products, do not look at its software often thousands of dollars but can be downloaded and generated for free, but there is no corresponding system evaluation and optimization performance comparison, there is a group of workstations, enterprise applications or industrial software test result numbers, PC OEM optimization product design and drivers are meaningless. SPEC's most entry-level product/test software is also geared towards PC workstations, and even DCC applications are just getting started, so it's not surprising that it's not taken seriously by individual consumers.
Spec's recently updated test software includes SPECViewperf 2020 v3.0, which will be featured later in our mobile workstation product testing, which will not be used in testing commercial PC products.
Compared with consumer PCs, the application scenarios of commercial PCs have clear requirements, highly multi-tasked and traceable processes, but users have clearer and clearer requirements for the performance of commercial PCs. Therefore, in the evaluation, there is also a high weight for the performance of "product compatibility/stability" and "performance stability/continuous output" of commercial PCs. At the same time, reflected in the test process of commercial test software, it is reflected in the multi-application parallelization and process sequence process with the goal of completing the project, not only all test software requires legalization and commercialization, but also the test cycle is long, the load fluctuation is large and frequent, and the consumer PC, high-performance PC (HEDT, non-HPC), workstation form a sharp contrast.