azkaban 安裝

在hadoop領域，常見工作流排程系統有：Oozie,Azkaban,Cascading,Hamake

環境

centos 6.5
JDK 8
azkaban 3.79
gradle-3.5

hostname	ip
vps138	192.168.1.138
vps155	192.168.1.155
vps156	192.168.1.156

下載下傳

$ cd /var/tmp
$ wget https://github.com/azkaban/azkaban/archive/3.79.0.tar.gz
$ tar -zxf 3.79.0.tar.gz
$ cd azkaban-3.79.0

//# 發郵件支援SSL
$ vi ./azkaban-common/src/main/java/azkaban/utils/EmailMessage.java
//# 加到import區域
import java.security.Security;

//# 修改sendEmail()方法
props.put("mail.smtp.ssl.trust", this._mailHost); #在這之後添加
Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
final String SSL_FACTORY = "javax.net.ssl.SSLSocketFactory";
props.setProperty("mail.smtp.socketFactory.class", SSL_FACTORY);
props.setProperty("mail.smtp.socketFactory.fallback", "false");
props.setProperty("mail.smtp.socketFactory.port", "465");

編譯

//# 安裝gradle
$ wget https://services.gradle.org/distributions/gradle-3.5-bin.zip
$ unzip gradle-3.5-bin.zip
$ mv gradle-3.5 /usr/local/gradle
$ echo 'export GRADLE_HOME=/usr/local/gradle' > /etc/profile.d/gradle.sh
$ vi /etc/profile.d/path.sh
export PATH=$PATH:$GRADLE_HOME/bin

$ source /etc/profile
//# 添加國内鏡像
$ mkdir -p /root/.gradle
$ vi /root/.gradle/init.gradle
allprojects{
    repositories {
        def ALIYUN_REPOSITORY_URL = 'http://maven.aliyun.com/nexus/content/groups/public'
        def ALIYUN_JCENTER_URL = 'http://maven.aliyun.com/nexus/content/repositories/jcenter'
        all { ArtifactRepository repo ->
            if(repo instanceof MavenArtifactRepository){
                def url = repo.url.toString()
                if (url.startsWith('https://repo1.maven.org/maven2')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_REPOSITORY_URL."
                    remove repo
                }
                if (url.startsWith('https://jcenter.bintray.com/')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_JCENTER_URL."
                    remove repo
                }
            }
        }
        maven {
            url ALIYUN_REPOSITORY_URL
            url ALIYUN_JCENTER_URL
        }
    }
}

$ cd /var/tmp/azkaban-3.79.0
$ gradle wrapper --gradle-distribution-url https://repo.gradle.org/gradle/dist-snapshots/gradle-kotlin-dsl-4.9-20180629133202+0000-all.zip
$ ./gradlew distTar -x test

複制build 後的tar包

編譯好的檔案都放在各目錄的build/distributions/下

$ cp ./*/build/distributions/*.tar.gz /var/tmp
az-hadoop-jobtype-plugin-*.tar.gz
az-hdfs-viewer-*.tar.gz
az-jobsummary-*.tar.gz
azkaban-db-*.tar.gz
azkaban-exec-server-*.tar.gz
azkaban-hadoop-security-plugin-*.tar.gz
azkaban-solo-server-*.tar.gz
azkaban-web-server-*.tar.gz
az-reportal-*.tar.gz

單機模式

不支援單機模式了,會報錯

azkaban.use.multiple.executors must be true. Single executor mode is not supported any more

分布式模式

安裝azkaban的web,exector服務

web和exector是分開的,多個exector在多台機器上

執行器(executor)通過資料庫mysql協調

這裡以vps138為web伺服器,

vps138,vps155,vps156為exector伺服器

$ mkdir -p /usr/local/azkaban

//# 在vps138上安裝web server
$ tar -zxf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-web-server-0.1.0-SNAPSHOT  /usr/local/azkaban/web

//# 在vps138上安裝executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec138

//# mysql建表語句,導入msyql
$ tar -zxf azkaban-db-0.1.0-SNAPSHOT.tar.gz

将編譯好的azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz傳到vps155,vps156;

//# 在vps155上安裝executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec155

//# 在vps156上安裝executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec156

配置web server

在vps138上配置web server

$ cd /usr/local/azkaban/web
$ mv conf/azkaban.properties conf/azkaban.properties.bak
$ vi conf/azkaban.properties
azkaban.name=BSS
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
velocity.dev.mode=false
lockdown.create.projects=false
lockdown.upload.projects=false
cache.directory=cache
jetty.connector.stats=true
executor.connector.stats=true
azkaban.jobtype.plugin.dir=plugins/jobtypes

# web界面端口設定
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8082

# 資料庫配置
database.type=mysql
mysql.port=63751
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban_password
mysql.numconnections=100

# 多executor排程規則設定
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

web預設使用者密碼

$ cat conf/azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

配置Executor

在vps138,vps155,vps156上配置

$ cd /usr/local/azkaban/exec
$ vi ./conf/azkaban.properties
executor.maxThreads=50
executor.flow.threads=30
executor.port=12321
azkaban.jobtype.plugin.dir=plugins/jobtypes
executor.global.properties=conf/global.properties

# 資料庫配置,在啟動executor時會向executors表插入記錄
database.type=mysql
mysql.port=63751
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban_password
mysql.numconnections=100

MySQL安裝

參見MySQL 安裝并修改密碼政策

建立azkaban需要的資料庫,使用者密碼,導入建表語句

mysql> create database azkaban DEFAULT CHARACTER SET utf8;
mysql> grant all on azkaban.* TO 'azkaban'@'%' IDENTIFIED BY 'azkaban_password';
mysql> use azkaban;
mysql> source /var/tmp/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql
//# 預設啟用executor
mysql> ALTER TABLE `executors` CHANGE `active` `active` TINYINT DEFAULT 1;

啟動azkaban服務

//# 會在執行目錄下生成日志
# 必須先啟動executor-server，否則web-server會報找不到executor的錯誤
$ cd /usr/local/azkaban/exec138
$ ./bin/start-exec.sh

$ cd /usr/local/azkaban/web
$ ./bin/start-web.sh

停止:
$ ./bin/shutdown-web.sh
$ ./bin/shutdown-exec.sh

WEB界面

http://ip:8082/

輸入: azkaban/azkaban

文檔位址: Azkaban documentation

azkaban 安裝

環境

下載下傳

編譯

複制build 後的tar包

單機模式

分布式模式

安裝azkaban的web,exector服務

配置web server

配置Executor

MySQL安裝

啟動azkaban服務

繼續閱讀

CentOs7 安裝redis4.0 遇到的坑一、CentOs7連接配接網絡二、make檔案的時候出錯三、安裝ruby的redis插件時候報錯最後

Thrift開發之 windows上安裝

（Operation not permitted）無法安裝ipython

AEM----的安裝

Android Studio 真機測試Unknown failure (Failure) Error while Installing APK

win7 localhost 配置 nginx ssl自簽名證書步驟報錯相關連結

Windows下Cygwin環境的Hadoop安裝（3）- 運作hadoop中的wordcount執行個體遇到的問題和解決方法

Testlink的安裝及使用

Apache httpd 安裝啟動demo（Window版）

apache httpd 配置

Ubuntu14.04 LTS下安裝mongodb

ubuntu14.04下安裝hbse1.0.1.1

艱難安裝LDAP,SSL認證

Apache配置SSLApache配置SSL

Windows下配置Apache的SSL服務

啟動azkaban時出現User xml file conf/azkaban-users.xml doesn't exist問題解決啟動azkaban時出現User xml file conf/azkaban-users.xml doesn't exist問題解決Azkaban Jetty server properties.