1.安装jdk
我安装的是
[[email protected] bin]$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
2.安装python
我安装的是
[[email protected] bin]$ python
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
3.storm下载
下载命令是
wget http://mirrors.hust.edu.cn/apache/storm/apache-storm-1.0.5/apache-storm-1.0.5.tar.gz
4.安装与搭建
[[email protected] soft]$ tar -zxvf apache-storm-1.0.5.tar.gz
修改配置文件
[[email protected] storm1.0.5]$ vi conf/storm.yaml
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "server-01"
- "server-02"
- "server-03"
#
nimbus.seeds: ["node0"]
#
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "org.apache.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
5.单机启动
启动nimbus
[[email protected] bin]$ ./storm nimbus&
[[email protected] bin]$ Running: /opt/java/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5/conf -Xmx1024m -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.nimbus
查看是否成功：
[[email protected] bin]$ jps
14066 Main
15987 nimbus
16119 Jps
启动UI：
[[email protected] bin]$ ./storm ui&
[[email protected] bin]$ jpRunning: /opt/java/bin/java -server -Ddaemon.name=ui -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5:/home/master/soft/storm1.0.5/conf -Xmx768m -Dlogfile.name=ui.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.ui.core
查看是否成功：
[[email protected] bin]$ jps
14066 Main
15987 nimbus
16277 Jps
16139 core
core就是ui，端口是8080
启动supervisor
[[email protected] bin]$ ./storm supervisor&
[[email protected] bin]$ Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
查看
[[email protected] bin]$ jps
16401 Jps
14066 Main
15987 nimbus
16297 Supervisor
16139 core
最后查看UI：
单机版完毕：
6.集群版搭建
10.0.2.10 node0 // 主机
10.0.2.15 server-01 //zk
10.0.2.6 server-02 //zk
10.0.2.5 server-03 //zk
10.0.2.7 kafka-01 //kafka
10.0.2.8 kafka-02 //kafka
10.0.2.9 kafka-03 //kafka
10.0.2.11 storm-1 //节点1
10.0.2.12 storm-2//节点2
10.0.2.13 storm-3//节点3
上面是我的机器的hosts 几个节点的配置文件统一为：
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "server-01"
- "server-02"
- "server-03"
#
nimbus.seeds: ["node0","storm-1","storm-2","storm-3"]
#
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "org.apache.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
然后node0之启动nimbus和ui，其他节点之启动supervisor
分别看看三个节点的启动情况： storm-1：
[[email protected] bin]# ./storm supervisor &
[[email protected] bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
[[email protected] bin]#
[[email protected] bin]#
[[email protected] bin]# jps
2420 Jps
2325 Supervisor
[[email protected] bin]#
storm-2：
[[email protected] bin]# ./storm supervisor &
[1] 2253
[[email protected] bin]# jps
2307 config_value
2318 Jps
[[email protected] bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
[[email protected] bin]#
[[email protected] bin]# jps
2358 Jps
2253 Supervisor
[[email protected] bin]#
storm-3：
[[email protected] bin]# ./storm supervisor &
[1] 2256
[[email protected] bin]# jps
2308 Jps
2297 config_value
[[email protected] bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
[[email protected] bin]#
[[email protected] bin]# jps
2256 Supervisor
2344 Jps
[[email protected] bin]#
三个节点都启动起来了
下来看看ui：
由于我没改三个storm的节点的hostname 所以分不出来具体是哪个