華為鲲鵬雲ARM64 CentOS部署Mesos叢集和Marathon

1. 在叢集所有主機上編輯

/etc/hosts

将叢集内所有主機的IP和hostname添加進去。

2. 搭建一個zookeeper叢集

這邊使用的docker-compose跑的zk，分别在三個節點上各跑一個組成叢集。docker-compose.yml（注意yml檔案中的ZOO_MY_ID，三個節點分别是1，2，3）：

version: '2'
services:
  zookeeper:
    image: arm64v8/zookeeper:3.3
    environment:
      ZOO_SERVERS: server.1=campusphere-kp02:2888:3888 server.2=campusphere-kp03:2888:3888 server.3=campusphere-kp04:2888:3888
      ZOO_MY_ID: 1
      ZOO_PORT: 2182
    network_mode: "host"
    restart: always
           

使用

docker-compose -f /opt/docker/docker-compose.yml up -d

啟動zk。

3. 叢集中找一台機器用來編譯mesos。

下載下傳mesos源碼包：http://mirrors.tuna.tsinghua.edu.cn/apache/mesos/1.8.1/mesos-1.8.1.tar.gz

4. 搭建jdk+maven環境

4.1. 下載下傳jdk和maven。jdk到oracle官網下載下傳， maven下載下傳位址：http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz

4.2. 解壓jdk和maven

tar -zxvf jdk-8u221-linux-arm64-vfp-hflt.tar.gz -C /usr/local
tar -zxvf apache-maven-3.6.1-bin.tar.gz -C /usr/local
           

4.3. 編輯

/etc/profile

，添加jdk和maven的環境變量

export MAVEN_HOME=/usr/local/apache-maven-3.6.1
export JAVA_HOME=/usr/local/jdk1.8.0_221
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH
           

4.4. 使環境變量生效

source /etc/profile

，執行

java -version

和

mvn -v

檢查環境是否正常。

5. 安裝依賴

yum install -y tar wget git
yum install cppunit-devel
yum groupinstall -y "Development Tools"
yum install -y python-devel java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
           

6.解壓mesos源碼包并編譯(編譯和安裝時間比較長)

mkdir -p /usr/local/mesos
tar -zxvf mesos-1.8.1.tar.gz -C /usr/local/mesos
cd /usr/local/mesos/mesos-1.8.1
./config --prefix=/usr/local/mesos
make -j6
make -j6 install
           

7. 編輯

/etc/profile

添加mesos的環境變量(叢集内所有的主機都添加)

export MESOS_HOME=/usr/local/mesos
export PATH=${PATH}:${MESOS_HOME}/sbin:${MESOS_HOME}/bin
           

8. 使環境變量生效

source /etc/profile
           

9. 配置mesos

cd /usr/local/mesos/etc/mesos
cp mesos-agent-env.sh.template mesos-agent-env.sh
cp mesos-master-env.sh.template mesos-master-env.sh
cp mesos-deploy-env.sh.template mesos-deploy-env.sh
cp mesos-slave-env.sh.template mesos-slave-env.sh
chmod +x *.sh
           

9.1 修改

mesos-master-env.sh

# This file contains environment variables that are passed to mesos-master.
# To get a description of all options run mesos-master --help; any option
# supported as a command-line option is also supported as an environment
# variable.

# Some options you're likely to want to set:
export MESOS_log_dir=/data/mesos/log
export MESOS_work_dir=/data/mesos/data
export MESOS_ZK=zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos
export MESOS_quorum=2

# 解釋：
# MESOS_log_dir：log輸出的目錄，如果不設定，預設不會産生log
# MESOS_worker_dir:存放中繼資料的目錄，如果對配置檔案修改後，可能需要删除該目錄下的檔案，然後再重新啟動。
# MESOS_ZK：Zookeeper的相關資訊，我這邊的zk配置的是域名，使用的bind搭建的域名伺服器
# MESOS_quorum：用于HA，根據master的個數設定：有1個master，MESOS_quorum=1；有3個master，MESOS_quorum=2；有5個master，MESOS_quorum=3。
           

9.2 修改

mesos-agent-env.sh

和

mesos-slave-env.sh

（這兩個檔案一樣）

# This file contains environment variables that are passed to mesos-agent.
# To get a description of all options run mesos-agent --help; any option
# supported as a command-line option is also supported as an environment
# variable.

# You must at least set MESOS_master.

# The mesos master URL to contact. Should be host:port for
# non-ZooKeeper based masters, otherwise a zk:// or file:// URL.
export MESOS_master=zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos

# Other options you're likely to want to set:
export MESOS_log_dir=/data/mesos/log
export MESOS_work_dir=/data/mesos/run
export MESOS_isolation=cgroups/cpu,cgroups/mem
export MESOS_containerizers=docker,mesos
export MESOS_switch_user=0
export MESOS_docker_config=/root/.docker/config.json

# 解釋
# MESOS_isolation：隔離機制
# MESOS_containerizers：可用的容器實作機制，包括 mesos、external、docker
# MESOS_switch_user：是否用送出任務的使用者身份來運作，預設為true
# MESOS_docker_config：用于倉庫認證
           

10. 在編譯mesos的機器上将編譯好的mesos打包，并傳到叢集内的其他機器上解壓

cd /usr/local
tar -zcvf mesos.tar.gz --exclude=mesos/mesos-1.8.1 mesos
scp mesos.tar.gz [email protected]:/usr/local/
           

11. 啟動mesos應用

11.1 叢集中的master節點上需要啟動master

mesos-daemon.sh mesos-master
           

11.2 叢集中的slave節點上啟動agent

mesos-daemon.sh mesos-agent
           

11.3 本地使用master-ip:5050即可以打開mesos頁面（先在本地把叢集主機資訊加入到hosts檔案中）

12. 安裝Marathon

12.1 下載下傳marathon：http://downloads.mesosphere.com/marathon/v1.1.1/marathon-1.1.1.tgz

12.2 解壓安裝

tar -zxvf marathon-1.1.1.tgz -C /usr/local
           

12.3 修改啟動腳本

cd /usr/local/marathon-1.1.1/bin
vi start
# 在腳本最前面聲明MESOS_NATIVE_JAVA_LIBRARY，加上如下語句：
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/lib/libmesos.so
           

12.4 啟動Marathon

nohup ./start --master zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos --zk zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/marathon --http_credentials wisedu:1qaz2wsx --event_subscriber http_callback &
           

12.5 本地使用ip:8080即可打開marathon頁面，使用啟動指令中的使用者名和密碼即可登入

13. 在Marathon上建立一個應用測試一下

點選“Create Application”，打開“JSON Mode”，将用于測試的應用的json貼進去，這邊用了一個tomcat的鏡像進行的測試：

{
  "id": "/tomcat-test",
  "cmd": null,
  "cpus": 0.2,
  "mem": 1024,
  "disk": 0,
  "instances": 2,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "campusphere-kp01:5000/tomcat-arm64v8:jdk8-corretto",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 0,
          "servicePort": 10001,
          "protocol": "tcp",
          "name": "tomcat",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "healthChecks": [
    {
      "path": "/",
      "protocol": "HTTP",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false
    }
  ],
  "portDefinitions": [
    {
      "port": 10001,
      "protocol": "tcp",
      "labels": {}
    }
  ]
}
           

測試成功：