天天看点
返回

kubernetes与calico整合kubernetes与calico整合

kubernetes与calico整合

说明

以前的kubernetes集群都是基于flannel搭建的，但应用系统所用的容器之间都能互访，存在一定的安全性，因calico在网络方面可支持策略，本文档为基于calico搭建kubernetes集群的记录

所有文件已经下载完成，并放置与我的github上 calico-kubernetes。

环境准备

  • 宿主机系统CentOS 7.1 64bit
  • virtualbox 5.0.14
  • vagrant 1.8.1
  • CoreOS alpha 928.0.0
  • kubernetes v1.1.7
  • calicoctl v0.15.0
  • calico v1.0
  • calico-ipam v1.0

安装

相关配置文件及组件下载完成后目录结构如下所示：

➜  coreos  tree
.
├── cloud-config
│   ├── calico
│   ├── calicoctl
│   ├── calico-ipam
│   ├── easy-rsa.tar.gz
│   ├── key.sh
│   ├── kube-apiserver
│   ├── kube-controller-manager
│   ├── kubectl
│   ├── kubelet
│   ├── kube-proxy
│   ├── kube-scheduler
│   ├── make-ca-cert.sh
│   ├── master-config.yaml
│   ├── master-config.yaml.tmpl
│   ├── network-environment
│   ├── node-config.yaml_calico-
│   ├── node-config.yaml_calico-
│   ├── node-config.yaml.tmpl
│   └── setup-network-environment
├── manifests
│   ├── busybox.yaml
│   ├── kube-ui-rc.yaml
│   ├── kube-ui-svc.yaml
│   └── skydns.yaml
├── synced_folders.yaml
└── Vagrantfile

必要二进制工具下载 

# 创建目录
mkdir cloud-config && cd cloud-config
## 下载calico相关组件
wget https://github.com/projectcalico/calico-containers/releases/download/v0/calicoctl
wget https://github.com/projectcalico/calico-cni/releases/download/v1/calico
wget https://github.com/projectcalico/calico-cni/releases/download/v1/calico-ipam

## 下载kubernetes相关组件
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kubectl
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kubelet
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kube-proxy
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kube-apiserver
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kube-controller-manager
wget http://storage.googleapis.com/kubernetes-release/release/v1/bin/linux/amd64/kube-scheduler

## 下载环境设置工具
wget https://github.com/kelseyhightower/setup-network-environment/releases/download//setup-network-environment

## 下载证书制作工具（也可以使用CoreOS系统自带的，本文档中不包含后续再更新）
wget https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz

cloud-init配置文件模板

目录中

master-config.yaml

、

node-config.yaml_calico-02

、

node-config.yaml_calico-03

为启动集群时根据

.tmpl

文件自动生成的配置文件

master cloud-init模板

~/cloud-config/master-config.yaml.tmpl 内容如下：

#cloud-config
---
write_files:
  # Network config file for the Calico CNI plugin.
  - path: /etc/cni/net.d/-calico.conf
    owner: root
    permissions: 0755
    content: |
      {
          "name": "calico-k8s-network",
          "type": "calico",
          "etcd_authority": "172.18.18.101:2379",
          "log_level": "info",
          "ipam": {
              "type": "calico-ipam"
          }
      }

  # Kubeconfig file.
  - path: /etc/kubernetes/worker-kubeconfig.yaml
    owner: root
    permissions: 0755
    content: |
      apiVersion: v1
      kind: Config
      clusters:
      - name: local
        cluster:
          server: http://172.18.18.101:8080
      users:
      - name: kubelet
      contexts:
      - context:
          cluster: local
          user: kubelet
        name: kubelet-context
      current-context: kubelet-context


hostname: __HOSTNAMT__
coreos:
  update:
    reboot-strategy: off
  etcd2:
    name: "etcdserver"
    listen-client-urls: http://0.0.0.0:2379
    advertise-client-urls: http://$private_ipv4:2379
    initial-cluster: etcdserver=http://$private_ipv4:2380
    initial-advertise-peer-urls: http://$private_ipv4:2380
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://0.0.0.0:2380
  fleet:
    metadata: "role=master"
    etcd_servers: "http://localhost:2379"
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: setup-network-environment.service
      command: start
      content: |
        [Unit]
        Description=Setup Network Environment
        Documentation=https://github.com/kelseyhightower/setup-network-environment
        Requires=network-online.target
        After=network-online.target
        [Service]
        ExecStartPre=-/usr/bin/chmod +x /opt/bin/setup-network-environment
        ExecStart=/opt/bin/setup-network-environment
        RemainAfterExit=yes
        Type=oneshot
        [Install]
        WantedBy=multi-user.target
    - name: kube-apiserver.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes API Server
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=etcd2.service
        After=etcd2.service
        [Service]
        ExecStart=/opt/bin/kube-apiserver \
        --allow-privileged=true \
        --etcd-servers=http://$private_ipv4:2379 \
        --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \
        --insecure-bind-address=... \
        --advertise-address=$private_ipv4 \
        --service-account-key-file=/srv/kubernetes/kubecfg.key \
        --tls-cert-file=/srv/kubernetes/server.cert \
        --tls-private-key-file=/srv/kubernetes/server.key \
        --service-cluster-ip-range=../ \
        --client-ca-file=/srv/kubernetes/ca.crt \
        --kubelet-https=true \
        --secure-port= \
        --runtime-config=extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/ingress=true \
        --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
    - name: kube-controller-manager.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Controller Manager
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=kube-apiserver.service
        After=kube-apiserver.service
        [Service]
        ExecStart=/opt/bin/kube-controller-manager \
        --master=$private_ipv4: \
        --service_account_private_key_file=/srv/kubernetes/kubecfg.key \
        --root-ca-file=/srv/kubernetes/ca.crt \
        --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
    - name: kube-scheduler.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Scheduler
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=kube-apiserver.service
        After=kube-apiserver.service
        [Service]
        ExecStart=/opt/bin/kube-scheduler --master=$private_ipv4:8080 --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target
    - name: calico-node.service
      command: start
      content: |
        [Unit]
        Description=calicoctl node
        After=docker.service
        Requires=docker.service

        [Service]
        #EnvironmentFile=/etc/network-environment
        User=root
        Environment="ETCD_AUTHORITY=127.0.0.1:2379"
        PermissionsStartOnly=true
        #ExecStartPre=/opt/bin/calicoctl pool add 192.168.0.0/16 --ipip --nat-outgoing
        ExecStart=/opt/bin/calicoctl node --ip=$private_ipv4 --detach=false
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: kubelet.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Kubelet
        Documentation=https://github.com/kubernetes/kubernetes
        After=docker.service
        Requires=docker.service

        [Service]
        ExecStart=/opt/bin/kubelet \
        --register-node=true \
        --pod-infra-container-image="shenshouer/pause:0.8.0" \
        --allow-privileged=true \
        --config=/opt/kubernetes/manifests \
        --cluster-dns=.. \
        --hostname-override=$private_ipv4 \
        --api-servers=http://localhost:8080 \
        --cluster-domain=cluster.local \
        --network-plugin-dir=/etc/cni/net.d \
        --network-plugin=cni \
        --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: kube-proxy.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Proxy
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=kubelet.service
        After=kubelet.service
        [Service]
        ExecStart=/opt/bin/kube-proxy \
        --master=http://$private_ipv4:8080 \
        --proxy-mode=iptables \
        --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target

~/cloud-config/node-config.yaml.tmpl 内容如下：

#cloud-config
---
write_files:
  # Network config file for the Calico CNI plugin.
  - path: /etc/cni/net.d/-calico.conf
    owner: root
    permissions: 0755
    content: |
      {
          "name": "calico-k8s-network",
          "type": "calico",
          "etcd_authority": "172.18.18.101:2379",
          "log_level": "info",
          "ipam": {
              "type": "calico-ipam"
          }
      }

  # Kubeconfig file.
  - path: /etc/kubernetes/worker-kubeconfig.yaml
    owner: root
    permissions: 0755
    content: |
      apiVersion: v1
      kind: Config
      clusters:
      - name: local
        cluster:
          server: http://172.18.18.101:8080
      users:
      - name: kubelet
      contexts:
      - context:
          cluster: local
          user: kubelet
        name: kubelet-context
      current-context: kubelet-context

hostname: __HOSTNAMT__
coreos:
  etcd2:
    proxy: on
    listen-client-urls: http://localhost:2379
    initial-cluster: etcdserver=http://172.18.18.101:2380
  fleet:
    metadata: "role=node"
    etcd_servers: "http://localhost:2379"
  update:
    reboot-strategy: off
  units:
    - name: etcd2.service
      command: start
    - name: fleet.service
      command: start
    - name: setup-network-environment.service
      command: start
      content: |
        [Unit]
        Description=Setup Network Environment
        Documentation=https://github.com/kelseyhightower/setup-network-environment
        Requires=network-online.target
        After=network-online.target
        [Service]
        ExecStartPre=-/usr/bin/chmod +x /opt/bin/setup-network-environment
        ExecStart=/opt/bin/setup-network-environment
        RemainAfterExit=yes
        Type=oneshot
        [Install]
        WantedBy=multi-user.target
    - name: calico-node.service
      command: start
      content: |
        [Unit]
        Description=calicoctl node
        After=docker.service
        Requires=docker.service

        [Service]
        #EnvironmentFile=/etc/network-environment
        User=root
        Environment=ETCD_AUTHORITY=172.18.18.101:2379
        PermissionsStartOnly=true
        ExecStart=/opt/bin/calicoctl node --ip=$private_ipv4 --detach=false
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: kubelet.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Kubelet
        Documentation=https://github.com/kubernetes/kubernetes
        After=docker.service
        Requires=docker.service

        [Service]
        ExecStart=/opt/bin/kubelet \
        --address=... \
        --allow-privileged=true \
        --cluster-dns=.. \
        --cluster-domain=cluster.local \
        --config=/opt/kubernetes/manifests \
        --hostname-override=$private_ipv4 \
        --api-servers=http://172.18.18.101:8080 \
        --pod-infra-container-image="shenshouer/pause:0.8.0" \
        --network-plugin-dir=/etc/cni/net.d \
        --network-plugin=cni \
        --logtostderr=true
        Restart=always
        RestartSec=10

        [Install]
        WantedBy=multi-user.target

    - name: kube-proxy.service
      command: start
      content: |
        [Unit]
        Description=Kubernetes Proxy
        Documentation=https://github.com/GoogleCloudPlatform/kubernetes
        Requires=kubelet.service
        After=kubelet.service
        [Service]
        ExecStart=/opt/bin/kube-proxy \
        --master=http://172.18.18.101:8080 \
        --proxy-mode=iptables \
        --logtostderr=true
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target

集群附件组件

文件夹

manifests

中为测试工具以及DNS、kube-ui等附件组件配置文件

当启动集群时，将自动复制到master节点的home目录中

Vagrantfile配置 

require 'fileutils'
require 'yaml'

# Size of the cluster created by Vagrant
num_instances=

# Read YAML file with mountpoint details
MOUNT_POINTS = YAML::load_file('synced_folders.yaml')

module OS
  def OS.windows?
    (/cygwin|mswin|mingw|bccwin|wince|emx/ =~ RUBY_PLATFORM) != nil
  end

  def OS.mac?
   (/darwin/ =~ RUBY_PLATFORM) != nil
  end

  def OS.unix?
    !OS.windows?
  end

  def OS.linux?
    OS.unix? and not OS.mac?
  end
end

# Change basename of the VM
instance_name_prefix="calico"

Vagrant.configure("2") do |config|
  # always use Vagrants insecure key
  config.ssh.insert_key = false
  # 指定创建集群vm所需的box
  config.vm.box = "coreos-alpha-928.0.0"

  config.vm.provider :virtualbox do |v|
    # On VirtualBox, we don't have guest additions or a functional vboxsf
    # in CoreOS, so tell Vagrant that so it can be smarter.
    v.check_guest_additions = false
    v.memory = 
    v.cpus = 
    v.functional_vboxsf     = false
  end

  # Set up each box
  (..num_instances).each do |i|
    vm_name = "%s-%02d" % [instance_name_prefix, i]
    config.vm.define vm_name do |host|
      host.vm.hostname = vm_name
    host.vm.synced_folder ".", "/vagrant", disabled: true
    # 挂载当前目录到虚拟机中的/vagrant目录，自动化部署
    begin
      MOUNT_POINTS.each do |mount|
        mount_options = ""
        disabled = false
        nfs =  true
        if mount['mount_options']
        mount_options = mount['mount_options']
        end
        if mount['disabled']
        disabled = mount['disabled']
        end
        if mount['nfs']
        nfs = mount['nfs']
        end
        if File.exist?(File.expand_path("#{mount['source']}"))
        if mount['destination']
          host.vm.synced_folder "#{mount['source']}", "#{mount['destination']}",
            id: "#{mount['name']}",
            disabled: disabled,
            mount_options: ["#{mount_options}"],
            nfs: nfs
        end
        end
      end
    rescue
    end
    # 指定虚拟机的ip地址范围
      ip = "172.18.18.#{i+}"
      host.vm.network :private_network, ip: ip

      host.vm.provision :shell, :inline => "/usr/bin/timedatectl set-timezone Asia/Shanghai ", :privileged => true
      # 自动将对应二进制文件复制到对应目录中
      host.vm.provision :shell, :inline => "chmod +x /vagrant/cloud-config/key.sh && /vagrant/cloud-config/key.sh ", :privileged => true
      #host.vm.provision :shell, :inline => "cp /vagrant/cloud-config/network-environment /etc/network-environment", :privileged => true
      # docker pull 相应必要镜像文件
      host.vm.provision :docker, images: ["busybox:latest", "shenshouer/pause:0.8.0", "calico/node:v0.15.0"]
      sedInplaceArg = OS.mac? ? " ''" : ""
      if i == 
        # Configure the master.
        system "cp cloud-config/master-config.yaml.tmpl cloud-config/master-config.yaml"
        system "sed -e 's|__HOSTNAMT__|#{vm_name}|g' -i#{sedInplaceArg} ./cloud-config/master-config.yaml"
        host.vm.provision :file, :source => "./manifests/skydns.yaml", :destination => "/home/core/skydns.yaml"
        host.vm.provision :file, :source => "./manifests/busybox.yaml", :destination => "/home/core/busybox.yaml"
        host.vm.provision :file, :source => "./cloud-config/master-config.yaml", :destination => "/tmp/vagrantfile-user-data"
        host.vm.provision :shell, :inline => "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", :privileged => true
      else
        # Configure a node.
        system "cp cloud-config/node-config.yaml.tmpl cloud-config/node-config.yaml_#{vm_name}"
        system "sed -e 's|__HOSTNAMT__|#{vm_name}|g' -i#{sedInplaceArg} ./cloud-config/node-config.yaml_#{vm_name}"
        host.vm.provision :file, :source => "./cloud-config/node-config.yaml_#{vm_name}", :destination => "/tmp/vagrantfile-user-data"
        host.vm.provision :shell, :inline => "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", :privileged => true
      end
    end
  end
end

部署附件组件

在Vagrantfile所在目录中执行

vagrant up

,启动过程因挂载宿主机目录到虚拟机会要求输入宿主机密码.

启动过程中会使用vm中的docker pull必要的镜像文件，等待自动化部署完成

部署 集群附加组件

在当前目录中执行

vagrant ssh calico-01

进入到master主机中:

[email protected] ~ $ ls
busybox.yaml  kube-ui-rc.yaml  kube-ui-svc.yaml  skydns.yaml

# 查看节点情况
[email protected] ~ $ kubectl get node
NAME            LABELS                                 STATUS    AGE
   kubernetes.io/hostname=   Ready     h
   kubernetes.io/hostname=   Ready     h
   kubernetes.io/hostname=   Ready     h

# 创建kube-system namespace，部署skydns rc,部署skydns svc：
kubectl create -f skydns.yaml

# 部署kube-ui
kubectl create -f kube-ui-rc.yaml
kubectl create -f kube-ui-svc.yaml


# 查看执行结果

[email protected] ~ $ kubectl get po -o wide --namespace=kube-system
NAME                READY     STATUS    RESTARTS   AGE       NODE
kube-dns-v9-qd8i3   /       Running             h        
kube-ui-v5-spea    /       Running             h        

# 验证DNS
# 部署busybox工具
kubectl create -f busybox.yaml

[email protected] ~ $ kubectl get po -o wide
NAME      READY     STATUS    RESTARTS   AGE       NODE
busybox   /       Running             h        

# 验证DNS
[email protected] ~ $ kubectl exec busybox -- nslookup kubernetes
Server:    
Address : 

Name:      kubernetes
Address : 

# 查看calico状态
[email protected] ~ $ calicoctl status
calico-node container is running. Status: Up  hours
Running felix version rc6

IPv4 BGP status
IP:     AS Number:  (inherited)
+---------------+-------------------+-------+----------+-------------+
|  Peer address |     Peer type     | State |  Since   |     Info    |
+---------------+-------------------+-------+----------+-------------+
|  | node-to-node mesh |   up  | :: | Established |
|  | node-to-node mesh |   up  | :: | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 address configured.

# 查看busybox容器地址分配，为192.168.0.0
[email protected] ~ $ kubectl describe po busybox
Name:               busybox
Namespace:          default
Image(s):           busybox
Node:               /
Start Time:         Tue,  Feb  :: +
Labels:             <none>
Status:             Running
Reason:
Message:
IP:             
Replication Controllers:    <none>
Containers:
  busybox:
    Container ID:       docker://ffad63169095e31816bd10e45270e2d6add39480f61a5370ba76d7e4c5dd86b
    Image:          busybox
    Image ID:           docker://b175bcb790231169e232739bd2172bded9669c25104a9b723999c5f366ed7543
    State:          Running
      Started:          Tue,  Feb  :: +
    Last Termination State: Terminated
      Reason:           Error
      Exit Code:        
      Started:          Tue,  Feb  :: +
      Finished:         Tue,  Feb  :: +
    Ready:          True
    Restart Count:      
    Environment Variables:
Conditions:
  Type      Status
  Ready     True
Volumes:
  default-token-u1ub:
    Type:   Secret (a secret that should populate this volume)
    SecretName: default-token-u1ub
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath           Reason  Message
  ───────── ────────    ─────   ────            ─────────────           ──────  ───────
  h        m        {kubelet } spec.containers{busybox}    Pulled  Container image "busybox" already present on machine
  m       m        {kubelet } spec.containers{busybox}    Created Created with docker id ffad6316909
  m       m        {kubelet } spec.containers{busybox}    Started Started with docker id ffad6316909

# 查看集群信息中的对外服务
cor[email protected] ~ $ kubectl cluster-info
Kubernetes master is running at http://localhost:
KubeDNS is running at http://localhost:/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at http://localhost:/api/v1/proxy/namespaces/kube-system/services/kube-ui

在宿主机上浏览器打开http://172.18.18.101:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui即可看到集群信息。

部署完成，验证也ok，开心地玩耍吧。

Docker kubernetes CoreOS etcd calico github centos 集群 文档 应用
上一篇: Springboot相关配置——lxd相关配置连接数据库spring boot的日期格式和图标设置
下一篇: CoreOS ISO方式安装1. 安装介质准备2. CoreOS配置文件cloud-config.yaml准备，文件内容如下（手动修改某些值）3. 搭建web服务器，把准备好的文件存放在web服务器上5. 从ISO启动，安装

继续阅读