The article is selected from Yunqi Strategic Reference, No. 1, 2022, by Han Hongyuan, Chief Architect of Alibaba Cloud's New Finance & Internet Business Unit.
In the new era, the development of open source is more and more vigorous, the relationship between open source and cloud is becoming more and more complex, the degree of coupling is getting higher and higher, the cloud is one of the best environments allowed by open source software, and it also plugs in a safe and efficient take-off engine for open source software.
This paper attempts to introduce the evolution process of software development and delivery methods from the history of enterprise software, combined with the development law of open source software, analyze the common misunderstandings of open source software development and use, discuss the business model and ecology of open source software, and the combination and mutual promotion of open source software and cloud in the cloud era.
In May 1997, Eric S. Raymond published The Cathedral and the Bazaar, known as the "open source bible," and the open source movement has since had its own declaration of independence.
Since the development of open source, from the emergence of individual heroes such as Linus Torvalds, Kenneth Lane Thompson, Richard Matthew Stallman, Eric Steven Raymond, to more and more enterprise organizations realizing the value of open source and joining open source, the open source model of open, equality, collaboration and sharing has gradually become a new generation of software development models.
Open source has become one of the leading forces of global software technology and industrial innovation, especially in the cloud era, open source covers the global scenario of software development.
First, why open source is difficult to avoid
Feature 1: The application breadth and depth of open source basic software are improved
Basic software is the "soul" of the computing industry. In the past few decades, with the vigorous development of open source, early open source software has focused on the application side, but important basic software, such as operating systems and databases, still adopts a closed and proprietary development model and business model.
Today, basic software has also evolved into an open source, open approach. 97% of software developers and 99% of enterprises use open source software, and the use of open source software for basic software, industrial software, and emerging platform software has become an important driving force for overtaking in curves. The "Open Source Ecology White Paper (2020)" of the Chinese Academy of Information and Communications Technology pointed out that in recent years, domestic enterprises have gradually focused on the layout of open source projects in the field of basic software, and a number of open source projects have emerged in the fields of operating systems, databases, middleware, etc., including many top open source projects of international foundations.
Feature 2: The value created by open source software is highlighted
Open source has become the driving force of global digital technology innovation, growing into a powerful technological innovation model, and from the initial software industry to hardware, chips, video, IoT, AI and other fields, open source business model is also gradually maturing. Today, new products are open source, new architectures are open source, new platforms are also open source, and even top research results are released in open source form. Open source has gradually become a crucial innovation direction for global scientific and technological progress.
Feature three: The state attaches unprecedented importance to open source software
In 2021, the 14th Five-Year Plan for National Economic and Social Development of the People's Republic of China and the Outline of Long-term Goals for 2035 clearly stated that it is necessary to "support the development of innovative consortiums such as digital technology open source communities, improve the open source intellectual property and legal system, and encourage enterprises to open software source code, hardware design and application services". There is no doubt that open source is supported by the national strategic level and will embark on the fast track of leapfrog development.
The General Office of the People's Bank of China, the Secretariat Bureau of the Central Cyberspace Administration, the General Office of the Ministry of Industry and Information Technology, the General Office of the Banking and Insurance Regulatory Commission, and the General Office of the Securities Regulatory Commission jointly issued the Opinions on Regulating the Application and Development of Open Source Technologies in the Financial Industry (hereinafter referred to as the Opinions). In recent years, open source technology has been widely used in various fields of the financial industry, playing an active role in promoting technological innovation and digital transformation of financial institutions, but also facing many challenges such as security and controllability. The promulgation of the "Opinions" will help standardize the rational application of open source technology by financial institutions, improve the level of application and independent controllability, and promote the healthy and sustainable development of open source technology.
Second, the development of software and the history of open source
In the early days of the computer industry, software existed as a supporting role in helping hardware function. Before the birth of Unix in the 1970s, there was no precedent for developing operating systems using high-level languages. Since the software is not universal, and the software itself does not exist as a separate sales entity, but only as a software development project, it is also common practice to deliver the project and deliver the source code. Customers need to maintain the operation of the system continuously, and the developer is not willing to assume the responsibility for the maintenance of the whole life cycle of the software.
With the birth of Unix and its implementation mainly in C, an open and universal operating system became possible, and the development and licensing of software became possible with the pursuit of compatibility and versatility of hardware and software. In the early days of Unix's development, due to the special attitude of its owner AT&T to avoid entering the software industry (due to the fact that the main telecommunications business was under severe antitrust pressure and was unwilling to add additional trouble), when licensed to users, it was provided in source code form and did not provide follow-up technical support, allowing users to correct problems in their own environment and port to new hardware platforms. The universities, governments, and business organizations that were authorized at this stage laid the foundation for the subsequent development of Unix, and also sowed the seeds of the subsequent Unix disputes and divisions.
In the 1980s, with the enhancement and popularization of computer versatility, general software that can run on a large number of computers gradually gave birth to the commercial software industry and continued to flourish. A large number of software companies have sprung up and many are still industry giants, such as Microsoft, SAP, Oracle, Adobe, etc. A feature of this era is that as the users of the computer expand from professionals to ordinary users, the delivery form of software has gradually become only to provide running programs and no longer provide source code, and the manufacturers are also sold to users with manufacturer-defined licenses, and manufacturers in addition to selling software, they also provide paid technical support services: upgrade software for customers, correct problems, and provide a running environment for new equipment. Consumers as users no longer have the right to modify and run the software, and the authorized use of the software and technical support services form an important business model in the commercial software industry.
In the field of personal computers, there has been no essential change in the way customers use software to this day. Since the vast majority of users are not software development professionals and there is no need to modify the system or application software themselves, this model is well adapted to user needs and can be sustained.
In the server field, the Unix camp originated from the early code distributed by AT&T, and SUN, HP, DEC, IBM, etc. developed their own Unix systems based on different hardware architectures, forming an ecological pattern of so-called open systems. The need for universal and compatibility has spawned a unified standard such as posix in the future, but the ambiguity of Unix's authorization and legal provisions and the attitude and ambition of subsequent holders have led to a large number of problems, and various systems derived from Unix have been changed to their own brands to develop separately, and there is also a lack of unified coordination with the Unix trunk, which has given WindowsNT a huge business opportunity and developed into today's Windows Server series operating system. These commercial operating systems are still only licensed to the end user at runtime and provide commercial technical support services.
In the 1980s, another important event in the software industry was the establishment of the Free Software Foundation (FSF), in academic and scientific research institutions, "hacking spirit" has always been the pursuit of many people, the pursuit of software freedom is based on the right to modify the source code. It has to be said here that the GNU (GNU is Not Unix) initiative initiated by Richard Stallman (whose various reputations do not affect the meaning and value of GNU and FSF) redeveloped most of the tool systems that implemented Unix, and distributed these software under the GPL license, the GPL mandated the continuous provision of source code for derivative software, and GNU implemented most of the common unix tool system, However, it is still impossible to get rid of the dependence on the core of the Unix operating system, until the development of the Linux core began by Linus Torvalds in 1991 to fill this gap, and the Linux core and peripheral tool system that are also licensed by the GPL have formed the Linux ecosystem, and a large number of universities have accumulated rich experience in the use and development of Unix. The active participation of scientific and commercial institutions has contributed to the rapid growth and maturity of Linux.
After the mid-1990s, the Internet began to flourish, and various Unix derivatives and Windows NT also supported this development process in its infancy.
With the explosive growth of user scale, cost has become a factor that cannot be ignored, especially many start-up Internet companies are not willing to spend valuable funds on expensive hardware equipment and rigid software licensing. At this stage, Intel has also gradually entered the ranks of server CPU providers from the CPU provider of the PC, and the scale advantage has also made its CPU capabilities gradually comparable to the RISCCPU of the server, which provides core capabilities for various Unix servers running open systems, Linux has been selected by many emerging Internet companies as the background service operation platform because of its advantages of not mandatory fees and open subsequent source code modifications. Widespread application and large-scale deployment have also helped Linux continue to develop rapidly, evolving into a server operating system that supports key businesses, at the same time, various open source systems on the operating system have also been widely used and continuously developed, gradually forming the general situation of today's Internet giants based on open source software and self-developed business software.
In the 21st century, with the widespread use of open source software in the Field of the Internet, combined with the gradual establishment of Intel's dominance in the CPU field and the continuous decline of various closed open systems, many commercial enterprises have also begun to accept open source operating systems and open source technology platform software, and the richness of application scenarios has further promoted the development of various open source software, forming a pattern of fierce competition today.
Third, the definition of open source software and the development of licensing system
Literally, open source refers to the source code of open software to users, but simply providing source code is not the current standard definition of "open source software", "open source software" refers to the software to provide source code to users and grant the freedom to modify and continue to evolve, so the difference in regulations on the redistribution of modified software has formed a different open source software licensing system.
At present, the licensing system of open source software is mainly divided into two categories: one is mandatory to re-open source, representative of which is the GPL licensing system; the other is not mandatory to derive open source, for the loose open source license, representative of the BSD, MIT, Apache and other licensing systems. The GPL licensing system mandates that the source code be provided at the time of downstream "distribution", and the compatibility of the GPL may not be changed, and the transition from open source software to non-open source software is restricted (but there is room for use in the definition of "distribution"). Forcing open source to continue has spawned different variants of licenses for service providers, especially cloud service providers, such as AGPL. Due to the excessive and complex restrictions, the acceptance of AGPLs is limited. A permissive open source license that requires a declaration of origin and retention of the license portion of the original software, but does not require a derivative portion to be mandatorily open source.
Specific criteria for judging open source software can be found in the 10 principles of the https://opensource.org/osd:
1. Do not restrict the freedom of redistribution of software;
2. Provide source code that can be used for continuous development;
3. Allow modification and development work on the basis of source code;
4. Ensure the integrity of the original source code, the change can be in the form of a patch, or you can change the name or version number to distinguish;
5. There are no differentiated restrictions for specific people or groups of people;
6. There is no restriction on the use of the software;
7. No stricter license restrictions are imposed when derivative products are issued;
8. The software license is effective at the level of each component and not only in the whole;
9. The software license does not spread to other software distributed in the same medium;
10. The software license is not accompanied by terms that affect technology neutrality.
Fourth, the impact of open source software on the business model of the software industry
1, open source does not change the basic mode of software operation
In a mature software market, the technical support of the software itself is equally important as the development and distribution itself, and no one has been able to prove the correctness of a complex software so far. The quality of the software depends on the design, implementation, testing and continuous repair of the exposed problems in the process of operation, open source software changes only the development stage and distribution form, other cycle software laws are still the same, especially for a large number of non-developers to use the basic platform software, completely the technical support work to the user is not feasible, technical support capabilities also directly affect the software can support the business, in line with the development of the software itself to support the customer's business operation and ensure service quality, It's still an important part of the software business.
2. Open source has become a model and promoter of distributed collaborative development model
Today's open source software, the more important meaning is a collaborative development model, open source software project governance methods also have closed and open differences, open source project dominance and management rights have two common ways, one is that the project belongs to a foundation organization, such as Apache, the other is the initiator or creator-led management model, such as MySQL and Linux core. It's hard to tell in a simple way which way is necessarily the best, but relatively speaking, being run by a reputable nonprofit with a mature institutional model is more likely to be trusted and attract a wider range of participants. Cross-organization and cross-regional collaboration is effectively supporting the continuous development of various software capabilities, and software with tens of millions of lines of code such as Linux core is still iterating rapidly, which also promotes the continuous rapid development of the entire IT industry.
3, open source does not mean free
Open source software originated from the free software movement, free software chooses the name of Free Software is often misunderstood as "free", in fact, here free means "free", open source is the embodiment of the spirit of software freedom. Today, even for software that promises to be completely open source, such as Linux, the main form of commercialization is still to provide quality-assured distributions and technical support services, and charging licensing fees is not an approved form of business. MySQL is a special case because it provides different software licenses at the same time, and the commercial and community versions of the software are essentially two products, but most of the code of the community version is shared under centralized control, and the fundamental reason is that all intellectual property rights related to MySQL belong to a single commercial entity.
4. Providing software in a distribution and providing services on the cloud are still the mainstream business models
At present, most users of software are not developers of the software itself. The complex and large scale of the system and platform software is neither necessary nor realistic to rely on users to solve problems and evolve continuously. In addition to providing individual software, software distributions and cloud services require overall validation testing and optimization efforts. These work to ensure quality and compatibility is not fundamentally different from the release of proprietary software that is not open source, and these work is indispensable for the software to run with the operating environment, continuous technical support is a necessary means of software operation, and recognizing the value of these work is also a choice to improve efficiency and ensure the operation of the system.
Fifth, the challenge of enterprises as open source users
According to the analysis of the components of the open source ecology by the Ministry of Industry and Information Technology, the open source ecology is composed of five elements and three links. Among them, open source projects are the center, relying on open source community collaboration to form software, hardware and other open source projects. It involves multiple parties such as open source contributors, open source users, open source operators, and open source service providers, including open source governance, open source operation, open source business layout, etc., relying on infrastructure such as code hosting platforms to meet the requirements of open source rules.
Most businesses exist as industry open sourcers and industry users, and the challenges they face are different.
1. The Challenges of IT Architecture Governance – The CIO's Perspective
Different enterprises have different ways of using open source software because of their different STAGEs of IT construction. But either way, ultimately, it is necessary to consider unified governance of IT architecture.
First, the entire business of the new enterprise is built with open source software from the start. Commonly founded after Internet companies and open source have reached a relatively mature stage, they generally do not form the habit of paying for software and software services. Moreover, such companies regard IT capabilities as their core competitiveness and build their own core business systems based on open source technology platforms. The platform technology team is fully capable of adopting open source and customized open source derivative products, and customization and adaptation are often undertaken by their own development teams. However, with the development of customized systems and the improvement of business importance, code changes and software upgrades are facing great challenges, forced to choose to continue to increase technical investment, or to avoid changes in a conservative state, in the long run, it will fall into a backward and rigid situation, technology upgrades face high complexity and affect the continuous operation of business.
Second, traditional enterprises partially replace commercial software with open source software. Commonly founded in large and medium-sized enterprises for a long time, long-term use of commercial software to enjoy software technology services, the pursuit of cost reduction and efficiency and technical control has spawned the demand for open source software, new technologies and new development methods also make new business development can not refuse to become the mainstream of platform-level technology open source software.
Third, go through the process of choosing and switching between open source and commercial software. Specific industries have chosen commercial software at a certain stage of development, re-embraced open source software and increased their own research and development investment after crossing the development steps, typically such as Alibaba, e-commerce business has high requirements for transaction consistency, mainly relying on relational databases to provide support capabilities. Taobao started when MySQL can help run through, after the rapid expansion of the scale rely on commercial database software to survive the rapid growth period, the subsequent development of more dependent on the support of distributed architecture, with the accumulation of technology and architectural capabilities, and finally with the distributed architecture to eliminate the dependence on commercial database products, the overall technology stack also adopts the open source system and sustainable development.
In view of the characteristics of open source software itself, the different version routes, and the complexity of different application open source combinations, different enterprises have the right to choose different enterprise-level technical architecture routes, but they all need to consider avoiding repeated wheels in the use of open source.
2. Autonomous and controllable challenges
There are roughly two understandings of the meaning of "autonomous". One refers to the use of software within the scope of sovereignty that manufacturers have full intellectual property control, not limited by the supply of foreign products and services, so the "main" here refers to the national subject; the other "main" refers to the enterprise itself, which can freely modify and use the software according to its own needs and ideas according to its own needs and ideas.
"Controllable" corresponds to a "card neck", whether it is open source community or commercially licensed software. Some companies want to stop sourcing commercial distributions and commercial services of open source software from now on. It is true that concentrating on their own research and development, their own control, in specific scenarios, abandoning external commercial dependence, starting a new stove, and making a set of completely independent products, such successful examples are there, and some companies have really done it, but this is after all a very small minority; objectively, this practice also requires a large number of excellent software engineers, and has accumulated a lot of experience in open source software to be able to achieve, if the software business is not the main business, there are also great challenges in the long run.
For most enterprises, in order to be autonomous and controllable, risk avoidance, it is necessary to have a moderate grasp of the software that runs and depends on, it is best to carefully study the code, architecture, and testing of open source software, even if not, it is necessary to master the key verification capabilities and debugging capabilities to a certain extent, combined with the software upgrade and deployment capabilities, to ensure that urgent problems can be dealt with. The trick here is to collaborate with the mainstream community of software products to quickly build their capabilities through learning and imitation. Therefore, not only should enterprises not cut off their connection with open source vendors, but on the contrary, the more likely they are to face risks, the more extensively they should establish a connection with the open source business ecosystem, unless this connection is really cut off by external objective irresistible reasons.
3. The challenge of controllable cost
The cost of most open source software is hidden, from the cost of ongoing quality control, upgrades, and operational services. Open source software itself is also a software product, will also follow the basic laws of software, the use of a large number of open source software combinations for technical support capabilities are very high requirements. Although the quality and reliability of widely used open source software are very high, it is very difficult to support the continuous operation of massive source code products by relying on the technical strength within a single enterprise, and only a few giants that regard IT as their core competitiveness are likely to provide very high stability guarantee for their business.
Although enterprises are increasing IT investment in the digital age, it is still necessary to avoid a cognitive misunderstanding: the licensing cost of a single software is not the whole investment, and the subsequent support investment is the highlight. Companies must move beyond a simple bottom-line cost model or ROI calculation logic to see overall costs from a business perspective.
4. The challenge of security
The rise in the number of open source software vulnerabilities can be attributed to the widespread adoption of open source components. The massive increase in the open source community over the past few years, as well as media coverage of some recent data breaches, has led to an increased focus on open source security. According to statistics, 84% of open source code repositories contain at least one vulnerability, and 60% of audited codebases contain high-risk vulnerabilities. In December 2021, the latest log4j2 remote code execution vulnerability is a typical event, the vulnerability has a huge impact, once exploited by an attacker, it will cause serious harm. Open source software security issues are mostly considered to stem from administrative oversight. It is mainly reflected in: first, the update of open source software users is not timely; second, open source projects lack processes and management tools such as SDL; third, attackers have increased attacks on the open source ecosystem.
5. The challenge of compliance
Taking the financial industry with the strictest compliance requirements as an example, the Opinions on Regulating the Application and Development of Open Source Technologies in the Financial Industry clearly states that financial institutions should follow the relevant laws and licensing requirements of open source technologies, use open source technologies in compliance, clarify the scope of use of open source technologies and the rights and obligations of use, and protect the legitimate rights and interests of open source technology authors or right holders.
Given the compliance of open source, the licensing model of source code becomes critical. In order to proactively identify problems, the initial stage can consider the use of centralized scanning, in the enterprise to use the software version of the batch scan before the release of the source code open source compliance. But with the increasing use of open source software, scanning itself is a complex process on the one hand, and leaving developers with too little time to fix problems once they are found. How to efficiently count the licensing and use of open source products in the enterprise, quickly correct the use of non-compliance, and form a list of enterprise open source software compliance is one of the major challenges faced by enterprises in using open source software compliance.
Sixth, the challenge of enterprises as open sourcers in the industry
1. The challenge of building an ecosystem
The early operation of the open source community did not reflect the actual application of open source technology in various industries and enterprises. At present, the operation of open source communities and foundations pays more attention to industrial communication and user sharing, and the cloud-based usage statistics of software products may form a new ecological operation mechanism.
2. The challenge of business model
The business model of open source products basically incorporates the following four models: open-core while providing differentiated business products; professional services (Professional Services); hosting (Hosting); market (market place), providing app stores, app markets, plug-ins.
For successful open source companies, the most common model is open core products, with hosting, services as a second and third source of revenue. These combinations may be valid for an open source product or an industry at some point. However, as the market competition and customers continue to change, these models may not be applicable, and you may need to find a unique business model for your products. The traditional offline software sales, distribution and delivery model is difficult to become an open source business model.
Seventh, the use of open source software by enterprises in the cloud era
1. Changes brought about by cloud computing
Customers are responsible for the entire use process from purchasing software to using the service directly, without paying attention to the supporting details outside the service. Reduced customer sensitivity to commercial terms related to the software itself, and opacity after service packaging also weakens user participation in software development, especially as the responsibility for ensuring operation and capability provision shifts to cloud providers.
Widely used open source software licenses were formulated earlier, when cloud computing had not yet appeared or had no major impact, and the license terms were mainly aimed at the situation where the software was mainly distributed. Cloud services, especially the software contained in the packaged PaaS service, circumvent the mandatory re-open source obligations such as the GPL, and the cloud services derived from open source to a certain extent connivance cloud vendors to reduce their contributions and sharing to the open source software itself.
The successful experience of Internet giants widely using open source software and the continuous development of public clouds have proved that closed-source infrastructure and support platform software in most scenarios no longer have a general development prospect.
2. The cloud and open source develop in synergy
No customer wants to be bound by a specific vendor, the same is true of the public cloud, the basic platform that supports application operation has evolved from the operating system and database of the software era to container scheduling, and the hybrid cloud will continue to develop as a support method that will exist for a long time. Based on these conditions, the cloud platform should provide a general open technology, in terms of the functional interface used by the application, comply with the versatility of the standard, do not limit the portability of the application; in the lower layer of the operating engine, provide differentiated non-functional support to help customers more effectively achieve business goals; combined with their own characteristics to build a key area of differentiated service capabilities, such as databases, etc., to provide users with agile and flexible development capabilities and stable and efficient support capabilities.
Mainstream cloud computing vendors are playing an increasing role in the enterprise IT market, and the effective integration of mainstream open source software with cloud platforms is an important capability embodiment of cloud platforms. Open source technology is also the future development direction of cloud services, and actively and effectively participating in mainstream open source projects is an important opportunity for cloud vendors. Large cloud vendors invest manpower and money to help open source projects continue to develop, which helps cloud technology capabilities be effectively utilized.
For users, the mainstream cloud platforms in the market provide much more functional consistency than customers build independently in terms of the main technical capabilities of containers, middleware and databases. When designing their own business systems, users should also fully consider the abstraction and management of different levels of the technology platform, and reduce the hard binding of specific management capabilities. It is not realistic for applications to switch between different cloud platforms without changing a line of code, but the pursuit of Yunte's ability not to invade the application code but only to play a role at the management level should be a realistic and feasible approach, which is also the direction of platform-neutral cloud native pursuit.
Public clouds and hybrid clouds of all forms are the mainstream bearer forms of itatry in the future. Open source software has proved itself to be an effective and mainstream development form of platform technology software, and the combination of cloud computing and open source will become an inevitable trend. Open source software runs on the cloud to varying degrees, providing technical capabilities and will be the main choice for enterprise IT in the future. Different degrees of hosting, the difference in the degree of management and cloud integration, the optimization ability of the runtime, will be the difference in cloud services, stronger management capabilities integration will often be accompanied by deeper operational optimization and less management requirements, the services provided will also have a stronger ability to operate autonomously, the result is PaaS services, such as databases, etc., customers will also choose to deploy and manage themselves, in this case, cloud infrastructure needs to provide enough openness for customers to integrate.
At the same time, the service capabilities of the infrastructure will also be directly reflected in the overall cost performance of the final service and business applications. Software, especially technology platform software, requires a lot of test verification and optimization adjustment to achieve the best stability and cost performance, and the public cloud operating environment and the massive and differentiated customer scenarios it supports provide the best environment for the maturity of the software. Large public cloud vendors are also the most important pool of resources capable of investing in software development validation and ongoing support services in the future. In addition to optimization, maintaining the openness and compatibility of the hosted software services themselves is a very important direction of effort, and attracting and retaining customers in a running effect rather than a black box will become the core competitiveness of the cloud platform.
The continuous development of cloud computing provides a new stage for open source software, and the cloud platform cooperates effectively with the open source community and avoids locking to gain the trust of users, which is an important embodiment of the openness of the cloud platform and will also be the supporting force for the sustainable development of the IT industry in the future.
Overall, the innate nature of the cloud and its ability to evolve since its inception have provided a natural ability to address the challenges posed by the sustainable use of open source software by enterprises.
Currently, the cloud is proven to be the best platform for enterprise IT architecture governance, including governance of open source technologies.
The cloud platform can better focus on solving security and compliance issues, and scan, vulnerability scanning, upgrades, etc. for different open source software compliance.
The development of cloud native has enabled the integration of different basic open source software and innovative open source software technologies, and the observability, development efficiency and management efficiency of open source software in the cloud era have been greatly improved.
Cloud vendors themselves are the biggest users of open source, best practices, and can help enterprises manage and shield the risk of trial and error.
The cloud platform is naturally a collaborative platform that allows open source contributors and open source users to communicate with each other on the cloud to drive development.
In the new era, the development of open source is more and more vigorous, the relationship between open source and the cloud is becoming more and more complex, the degree of coupling is getting higher and higher, the cloud is one of the best environments allowed by open source software, plugging in a safe and efficient take-off engine for open source software.