during
the 18th techday of the computing conference, qi jun,
cto of router software company limited from nanjing, made a presentation titled
how can medium-sized and small enterprises make clever use of container
technology, sharing the company's experiences, issues and lessons learnt during
its use of alibaba cloud container services and docker. focusing on
the impact of container technology on business production and the overall
productivity, the presentation is of significant reference to medium-sized and
small enterprises.
download presentation slides
the
following is a summary of the shared points of view at the activity.
<b>dilemmas</b>
first, let's take a look at the first
dilemma:
there
are too many or overlapping sub-products in the product line, leading to
numerous repeated basic services, such as e-mail services and sms services. but
in development stage, they cannot be split.
second dilemma is the o&m. at present, there are a total of 40 servers in
the company and a bunch of services, but only one full-time o&m personnel.
third dilemma is the core issue: extremely low efficiency in deliveries of
products/services, giving rise to a serious waste of human resources.
fourth dilemma is being unable to effectively manage load peaks and valleys for
reasonable resource allocation.
fifth dilemma is the demanding requirements of the company on reliability and
security. every day, hundreds of media rely on our products for their
publication and production. if we cannot improve business reliability, a
problem means the loss of an issue of a newspaper or an episode of a tv program
in a region.
<b>old architecture</b>
above figure illustrates the old architecture of the company, and every line is
identical. a heap of clients visit a public ip address
behind which may be an ecs server. on the ecs server, one
or more applications may be deployed, and these applications are connected to
alibaba cloud database services, or cache or log services. all of these
constitute the minimum unit. during peak hours, there may be up to 60 minimum units
in the company, involving nearly 80 ecs servers, leading to upgrading and
maintenance issues. but this approach has to be abandoned. because once the
code is released, you have to negotiate a time with the client for upgrading.
but clients deploy their systems independently, and it is impossible to agree
on a uniform upgrading time for all clients.
<b>present architecture</b>
in
the current architecture, users access through the internet, first by way of
the slb (load balancing forwarder) which can be
understood as an official website ip address that
accesses its back-end servers through the official ip addresses provided by slb; in the
back-end server, we have a vpc network which houses around 20-30 ecs servers.
the 20-30 ecs servers constitute the present four
major clusters in use. the container cluster needs to have several container
instances to run after the cluster is created.
upon receipt of the request for access, the container cluster
will forward the requests to applications according to the request domain name
and port number. after the requests arrive at the application, the containers
for various services on the application will connect to alibaba cloud databases
or self-commending databases.
<b>core issues</b>
reliability, cost and agility are
three core issues. agility refers to agile development and deployment; cost is a
shared concern of the boss and the cto; and
reliability secures our survival.
<b>container technology & agile development</b>
container
technology and agility development now face three problems:
first,
the uniform development, testing and business environments. the difference in
versions, applications and operating environments may cause a variety of, and
usually unexpected, problems. meanwhile, new employees still need to repeatedly
establish various development, testing and operation environments, lowering the
efficiency.
for
this issue, our solution is to encapsulate and package all the basic
environments in the image warehouse. new employees only need to download the
image, and put his/her own code into the image during local debugging and
development. although read-only during the process, the image is actually
connectible with data on external disks. in addition, the developed and debugged
code won't suffer problems, because all the images are consistent and are
operating-system-independent.
second,
how can we achieve continuous development of applications developed during
different stages. this is a pain point for development teams of many
medium-sized and small enterprises. new employees are reluctant to maintain and
fix issues in the old code.
such problems, we can further break down the application: several items of a
big application can be separated to small applications, slashing the cost for
maintenance. during continuous development, these small applications do not
require re-writing and you can focus on the necessary tasks.
third,
code contamination and manual packaging faults need urgent solution. when a
company has many small applications, packaging may see frequent issues.
currently,
we adopt git auto build to solve the problem, because
it gets code from git. when we push a branch, an
image will be automatically created, in which case code contamination and
manual packaging faults are unlikely to happen.
<b>container technology & business cost</b>
next,
let's take a look at the relationship between container technology and business
costs. at present, there are several cost problems as follows:
long
downtime is costly for updating and greatly impairs user experience.
previously, we could only update at 3 o'clock in the morning. if any
problems occur, we needed to solve them before 8 o'clock. this has
increased the cost a lot, and sometimes the code updating will take a whole
night, let alone the more effort-consuming case when rollback is required.
for this issue, the blue-green release mentioned in the speech of the
previous presenter actually is a good solution.
simultaneous
updating for multiple loaded servers is hard to achieve, as the rollback
cost in case of errors is very high. blue-green release can solve this
problem. in our company, we adopt alibaba cloud containers that can
accommodate minuses. before the container is fully updated, users cannot
access the content in the new image.
business
precision elasticity is not available, and server-level elasticity is far
from enough. among our 40
servers, probably only one third of their service time witnesses a
utilization rate of 80% or
above. during the remaining two thirds of their
service time, the resources are idle and wasted. through
container level elasticity, we can activate more containers for web
services or api services
to realize better loads and execution efficiency. in such circumstances,
more physical servers can provide such reliable computing resources and
the performance is far better than imagined. because in normal cases, it
is impossible for a server to always stay fully loaded under any
circumstances.
<b>container technology & reliability</b>
now
let's talk about container technology and reliability. one truth about
reliability is that on-cloud full failover may not solve the problem instantly,
but it is needed by all. on the cloud, many of us may have overlooked the hot
backup issue, thinking hardware faults are not likely to happen on the cloud.
but the truth is the other way round. alibaba's container service can be
configured through arguments. when a cluster fails, you can manage to put the
container into another cluster. although there are no perfect solutions for
domain name or port configurations, and manual adjustments are required, it is
great progress.
data
synchronization and sharing in multi-server load scenarios. the appendixes of
some old services cannot be separated in case of poor coupling, or some
concentrated reads or writes may exist. in such cases, shared storage should be
used to solve this issue. at present, we solve this problem through the two
solutions provided by the container service: first, the oss data volume,
and the other is nsa data volume. both can
support access from multiple containers to the same file data source, as well
as real-time concentrated writes and reads.
<b>the traps we once fell into</b>
now i want
to summarize the lessons i learnt over the years, in a hope they can be of some
help to you:
first, containers without decoupling are difficult to use. this
is much in evidence. because if the container has a bunch of applications with
high relevance, once the container goes wrong, the applications will fail and
the whole business system may even collapse. for this reason alone, we need to
decouple applications, which is very important.
second, it is the micro services. microservices are booming at
present, but it does not mean the architecture would become amazing after it is
split into microservices. microservices have their own merits, but again, it
does not mean all businesses should adopt microservices. only universal,
repeated and reusable applications are suitable to be split into microservices.
third, the bigger the project, the more difficult the container
architecture for use. a great majority of our projects are currently hosted in
container services, but not all of them. some products are too big to be placed
in a container service. this involves the internal management, as well as the
product or business scenario and user requests. the bigger the project, the
wider the scope, and the harder to impose a sweeping approach on it, that is,
placing the project into a container cluster.
fourth, reliability. do not count on docker alone to solve all the reliability issues. docker
and container technology, in my opinion, are both a kind of architecture
instead of a tool. reliability is related with a lot of elements, from the
network environment and the overall architecture layer, to the quality of
developers. these elements are beyond the control of docker.
finally, container technology is only a kind of architecture.
running docker
on a single service is an experiment. only when it drives the operation of a
cluster can it truly give play to its power.
<b>common scenarios</b>
the first
scenario is efficient api cluster. sometimes
we can encounter such circumstances that some apis in the
company are for external use, and some others are not, but the api can be used by the company app and the app may access the api
through the official approach, and the api can be called by internal services at the same time. but the
issue is: when an external domain name visits it, there are no problems. but
when an intranet domain name does so, dns
is required and some official traffic may be even consumed for the visit. as a
matter of fact, the server called through intranet may be very close to the api server.
in such circumstances, we can solve the problem using a model:
when an internal server initiates the call, use the intranet slb (alibaba's
intranet slb is free of charge); when an external server initiates the call,
use the internet slb (traffic charges only, if i recall it right); the two
container clusters (container clusters are alibaba's container services)
connected to the api
respectively are correctly configured internally. one advantage of this is:
when this service is used in a concentrated way, you can make choices at will.
on-cloud calls can be completed through intranet slb; and external calls can be completed through internet slb, facilitating a maximum and most
reliable access efficiency. regarding internet access, the response may be
around 10
milliseconds, while the figure for intranet access may be around 1 to 2 milliseconds.
the second
scenario is fast delivery of applications/services. our company focuses on saas services as well as applications and
software. most of the work is about assisting in development and customer
services, and some deployment is also undertaken by developers. we solve the
problem with a model. after the product image is developed, we submit the image
to the code library; o&m personnel enter the container service and create a
cluster, then they create an application using the orchestration template.
after the application is created, the service configurations are modified.
because the container is not running in the same way as the code we are
familiar with, some configuration files need to be modified. the current solution
is: we put all configurations in the container environment variables, which is
also the mainstream approach. then we modify the service configuration, the
environment variable and restart the service. after the service is restarted,
the restarted container will read an environment variable as its configuration
and it will then be running successfully.
<b>summary</b>
simply put: in the past year, from pains and the verge of giving
up to the flush of dawn, and to the bumps along the way, we should thank all
the media clients and alibaba for their constant support.