What would you do if you were to dominate the functional iterations of a product with tens of millions, or even hundreds of millions of users?
The challenges you need to face may come from new product demands brought about by changes in business strategy, and any changes to products, even if only interface adjustments, will be "reviewed" by countless existing users.
At this time, as the product owner, will you choose stability over everything, or self-innovation to pursue user and market value?
Through the research of Internet giants such as Facebook and Twitter, the author tries to peek into the secret of their "stable" iteration in the rapidly changing market - progressive delivery, and further explore how to use Spinnaker to achieve fully automatic progressive delivery.
In this article, you will learn:
1. What is progressive delivery.
2. Why progressive delivery gives products in large-scale organizations the ability to deliver continuously and iterate stably.
3. Practical experience for both small and large projects.
01
<h1 class="pgc-h-arrow-right" data-track="11" > what is progressive delivery</h1>
The mobile Internet era has given birth to a large number of giant Internet enterprises and projects, the technical complexity and organizational complexity of some large-scale projects are even no less than traditional industrial projects, in order to achieve the management and iteration of these projects, we try to turn our attention to the traditional industrial fields that have completed the industrial revolution to find answers.
The term "progressive delivery" originated from large, complex industrialized projects, which attempt to disassemble complex projects in stages and reduce delivery costs and time through continuous small closed-loop iterations.
According to the data, the term "progressive delivery" became popular in the Internet field after kubernetes and the concept of cloud life were popularized. Especially after the advent of continuous deployment pipelines, progressive delivery provides the infrastructure and implementation methods for Internet applications.
In the process of product iteration, the specific behavior of progressive delivery can be attached to the pipeline, and the entire delivery line can be regarded as a process of product iteration and a progressive delivery cycle. Progressive delivery is implemented in practice by technical means such as A/B testing and canary/grayscale release.
In facebook, for example, every time it releases a major feature, it goes through a typical gradual delivery process:
1. Iterative release
2. All employees of the company conduct A/B testing
3. A/B testing for specific users
4. Grayscale release
5. Full release
In the process of gradual delivery, the A/B testing link and the grayscale release link can decide whether to release all according to user data and market feedback, which can ensure iterative agility and market security.
(1) A/B test
You can A/B test the geolocation and gender combination criteria in the user profile to give them access to the new version, while other users continue to access the old version. After some time, study user behavior data and user experience reports to decide whether to move on to the next release.
(2) Canary/Grayscale release
Specific shunting techniques can be used to make traffic shared by the new and old versions, such as the typical MurmurHash algorithm.
02
<h1 class="pgc-h-arrow-right" data-track="29" > technical and commercial value</h1>
In principle, these techniques are not very new, such as A/B testing, which we can achieve in the most primitive way (business code adds logical judgment conditions), but why is it not used on a large scale?
The reason is simple: the implementation of pure business code depends on technology, and the demand side cannot control the environment and conditions of A/B testing, and this over-reliant on technology development cannot be used at scale.
What we need is an implementation that is completely out of line with business code, preferably in an automated/semi-automated way, and trying to incorporate this action into existing internal processes.
Now, with Spinnaker, a cloud-native continuous deployment tool, automated, progressive delivery is possible. We designed the Spinnaker Pipeline, which we used to refer to How Facebook published it.
It mainly implements the following functions:
After the pipeline is triggered, the K8S Job Migrate database is automatically executed and the new version is deployed to the pre-release environment;
Manually confirm whether A/B testing is carried out before releasing the production environment;
After the A/B test is passed, set the scale of grayscale release and perform automatic grayscale release;
Manually confirm whether it is fully released to the production environment;
Automatic configuration of current limiting and circuit breaker strategies to ensure stable production.
For developers, this progressive delivery goes through multiple rounds of grayscale releases, A/B testing, minimizing code bugs for release to production.
For O&M personnel, this almost automatic delivery method changes the tradition of manually modifying yaml files and manually applying, which can greatly reduce the human error generated by the release and reduce the communication cost with the development through automatic triggering.
For product managers and operations personnel, product iteration is no longer based on internal team guessing, but based on actual user experience data, understanding the feedback of new features to users and the market can minimize the market risk of new features.
03
< h1 class="pgc-h-arrow-right" data-track="47" > the core principle</h1>
In the example above, we used Traefik as the cluster gateway and Router to match Host dev.coding and pro.coding, so that traffic is distributed differently according to different release stages.
(1) Dev environment architecture diagram
When dev.coding is accessed, Router matches this Host rule and forwards traffic to a service called k8s-flask-nodeport (i.e., the service of the Dev environment).
The core configuration code for the Traefik Router is as follows:
(2) A/B test environment architecture diagram
When accessing pro.coding, router matches this Host rule, checks if header matches, and will decide whether to forward traffic to the services of the k8s-flask-canary or k8s-flask environment based on the matching result.
The core configuration code for A/B testing the Traefik Router is as follows:
(3) Grayscale release architecture diagram
When accessing pro.coding, Router matches this Host rule and forwards traffic proportionally to the k8s-flask-canary or k8s-flask service based on the configured Weight weights.
(4) Circuit breaker and current limiting architecture diagram
In the production environment, we generally use current limiting and fuse technology to cope with traffic surges, sacrificing the experience of some users to ensure the stability of the production environment. Traefik internal fuse and current limiting are achieved by configuring middlewares, matching the traffic, and then confirming the secondary traffic of the middleware.
The core configuration code for Traefik Middlewares throttling is as follows:
The Traefik Middlewares circuit breaker core configuration code is as follows:
04
<h1 class="pgc-h-arrow-right" data-track="63" > summary</h1>
The advent of Kubernetes and Service Mesh has brought more possibilities for continuous deployment, and progressive delivery is just a more typical way of publishing that is formed with its convenience.
With the help of Traefik as a cluster gateway, we implemented A/B testing and grayscale release through shunt technology, and orchestrated these declarative profiles with Spinnaker to achieve a progressive delivery process.
The circuit breaker and current limiting capabilities provided by Traefik, combined with Spinnaker Pipeline's Webhook triggering and monitoring system (such as Prometheus), can automatically trigger circuit breakers and limit pipelines when the business system is under high pressure, changing the current limiting strategy and ensuring the stability of the production environment.
Of course, you can also introduce Service Mesh, which uses Istio to manage cluster traffic, achieving the same effect with Virtual Service and Destination Rule.
Spinnaker: The Future of Continuous Deployment in Cloud-Native Multi-Cloud Environments! For more technical details about Spinnaker, read Spinnaker in Action: A Continuous Deployment Approach for Cloud-Native Multi-Cloud Environments.
▊Spinnaker in Action: Continuous Deployment solutions for cloud-native multi-cloud environments
Wang Wei, Wang Zhenwei
It sets a precedent for Spinnaker, a continuous deployment tool for cloud-native multi-environments, and is explained in depth
The case is based on the actual work of the front-line engineers of the large factory, which has very good guidance and practice
Provide rich image resources and practical source code to help readers get started quickly
This book focuses on continuous deployment scenarios for cloud-native and multi-cloud environments in 13 chapters, covering a declarative continuous deployment overview, Spinnaker basics and practices, canary release and grayscale release, deployment security, chaotic engineering, and production recommendations.
In terms of continuous deployment best practices, this book focuses on how to implement grayscale publishing, automated canary analysis, and chaos engineering, which are the core technologies for Netflix to achieve rapid and stable iteration. On how to land spinnaker, this book provides guiding opinions for the migration team from the perspective of people and organizational structure, and solves the problem of difficult landing of new technologies.