hbase（分布式nosql資料庫）安裝

1.修改/etc/hosts 檔案，添加以下主機資訊

172.20.10.12 hbase01

172.20.10.13 hbase02

172.20.10.14 hbase03

2.SSH 配置（互信配置）

systemctl disable firewalld.service

分别在 hbase01 hbase02 hbase03 上執行，一直回車就好，生成秘鑰

ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa

在hbase01上執行

ssh hbase01 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh hbase02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh hbase03 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys hbase02:~/.ssh/

scp ~/.ssh/authorized_keys hbase03:~/.ssh/

for i in {12..14};do scp -r ~/.ssh/id_rsa.pub [email protected].$i:~/.ssh/authorized_keys;done（簡捷指令）

驗證互信，各節點執行下面指令，能不輸入密碼顯示時間，配置成功

ssh hbase01 date;ssh hbase02 date;ssh hbase03 date;

3.同步時間

yum -y install ntpdate

ntpdate cn.pool.ntp.org

4.關閉 SELinux ,将 SELINUX= enforcing 更改為 disabled，并重新開機

vim /etc/selinux/config

4.安裝配置Hadoop

4.1 下載下傳Hadoop 到 /opt/ 目錄下 (Hbase01上一台下載下傳就可)

wget http://archive.apache.org/dist/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

tar xf hadoop-2.7.7.tar.gz -C /usr/local/

4.2 修改配置檔案 hadoop-env.sh 配置 hadoop 環境變量

export JAVA_HOME=/usr/local/jdk

export HADOOP_HOME=/usr/local/hadoop

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

4.3 core-site.xml 配置HDFS

<name>fs.defaultFS</name>

<value>hdfs://hbase01:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/data/hadoop/tmp</value>

</property>

</configuration>

mkdir -p /home/data/hadoop

4.4 hdfs-site.xml 配置namenode

<name>dfs.namenode.http-address</name>

<value>hbase01:50070</value>

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>hbase02:50090</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>/home/data/hadoop/name</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/data/hadoop/datanode</value>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

mkdir -p /home/data/hadoop/tmp

mkdir -p /home/data/hadoop/name

mkdir -p /home/data/hadoop/datanode

4.5 mapred-site.xml 配置架構

cp mapred-site.xml.template mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.application.classpath</name>

<value>

/usr/local/hadoop/etc/hadoop,

/usr/local/hadoop/share/hadoop/common/*,

/usr/local/hadoop/share/hadoop/common/lib/*,

/usr/local/hadoop/share/hadoop/hdfs/*,

/usr/local/hadoop/share/hadoop/hdfs/lib/*,

/usr/local/hadoop/share/hadoop/mapreduce/*,

/usr/local/hadoop/share/hadoop/mapreduce/lib/*,

/usr/local/hadoop/share/hadoop/yarn/*,

/usr/local/hadoop/share/hadoop/yarn/lib/*

</value>

</property>

</configuration>

4.6 yarn-site.xml 配置resourcemanager

<name>yarn.resourcemanager.hostname</name>

<value>hbase01</value>

</property>

<description>The http address of the RM web application.</description>

<name>yarn.resourcemanager.webapp.address</name>

<value>${yarn.resourcemanager.hostname}:8088</value>

</property>

<description>The address of the applications manager interface in the RM.</description>

<name>yarn.resourcemanager.address</name>

<value>${yarn.resourcemanager.hostname}:8032</value>

</property>

<description>The address of the scheduler interface.</description>

<name>yarn.resourcemanager.scheduler.address</name>

<value>${yarn.resourcemanager.hostname}:8030</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>${yarn.resourcemanager.hostname}:8031</value>

</property>

<description>The address of the RM admin interface.</description>

<name>yarn.resourcemanager.admin.address</name>

<value>${yarn.resourcemanager.hostname}:8033</value>

</property>

</configuration>

4.7 修改 Slaves

hbase02

hbase03

在叢集内所有機器上都進行建立，也可以複制檔案夾

for i in {hadoop02..hadoop03};do scp -r /home/data root@$i:/home/;done

for i in {hadoop02..hadoop03};do scp -r /usr/local/hadoop-2.7.7 root@$i:/usr/local/;done

4.8 配置環境變量

vim /etc/profile.d/hadoop.sh

export HADOOP_HOME=/usr/local/hadoop

PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

生效

source /etc/profile.d/hadoop.sh

4.9 叢集初始化

1. 首次啟動需要初始化HDFS(在master節點做一次就可以了)

hdfs namenode -format

第二次啟動，配置好 Hadoop 環境變量

start-all.sh

stop-all.sh

2.驗證成功

驗證 (再次提醒，防火牆一定要放開端口或直接關閉)

windows 主機通路網頁時，應在windows hosts 上加主機名ip 映射，或者直接通路

http://hbase02:50090/

http://hbase01:50070/

jps 指令檢視Linux 主機

[root@SNAPP02 hadoop]# jps

25282 NameNode

25576 ResourceManager

26092 Jps

[root@SNAPP03 hadoop]# jps

13521 SecondaryNameNode

13601 NodeManager

13413 DataNode

13992 Jps

[root@SNAPP04 hadoop]# jps

13729 NodeManager

14074 Jps

13611 DataNode

////////////////////////////////////////////////////////////////////////////////////////////////////////

5. Hbase 部署

工作需要，作者需要用的Hbase 版本為 1.4.x

5.1 下載下傳解壓

wget https://mirrors.bfsu.edu.cn/apache/hbase/1.4.13/hbase-1.4.13-bin.tar.gz

tar xf hbase-1.4.13-bin.tar.gz -C /usr/local/

5.2 配置 Hbase 環境變量

/usr/local/hbase-1.4.13/conf 目錄下修改 hbase-env.sh 添加

export JAVA_HOME=/usr/local/jdk

export HBASE_CLASSPATH=/usr/local/hbase

5.3 hbase-site.xml 配置hbase

mkdir -p /home/data/hbase/zookeeper

mkdir -p /home/data/hbase/zookeeper/data

mkdir -p /home/data/hbase/tmp

<name>hbase.rootdir</name>

<value>hdfs://hbase01:9000/hbase</value>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.zookeeper.quorum</name>

<value>hbase01,hbase02,hbase03</value>

</property>

<name>hbase.tmp.dir</name>

<value>/home/data/hbase/tmp</value>

</property>

<name>hbase.zookeeper.property.dataDir</name>

<value>/home/data/hbase/zookeeper/data</value>

</property>

</configuration>

指定叢集節點

vim regionservers

hbase01

hbase02

hbase03

将 Hbase 檔案夾複制到另外兩台伺服器

scp -r ./hbase-1.4.13/ hbase02:/usr/local/

scp -r ./hbase-1.4.13/ hbase03:/usr/local/

6. 啟動內建hbase

隻需要在hbase01 上啟動即可

啟動 [clsn@hadoop01 /usr/local/hbase/bin]

start-hbase.sh

停止

stop-hbase.sh

[root@SNAPP02 bin]# jps

25282 NameNode

28550 HMaster

25576 ResourceManager

28444 HQuorumPeer

29117 Jps

28703 HRegionServer

[root@SNAPP03 local]# jps

13521 SecondaryNameNode

13601 NodeManager

13413 DataNode

16246 HQuorumPeer

16365 HRegionServer

16654 Jps

[root@SNAPP04 local]# jps

13729 NodeManager

15970 HQuorumPeer

16089 HRegionServer

13611 DataNode

16379 Jps

//////////////////////////////////////////////////////

業務資訊

/////////////////////////////////////////////////////////////////////////////////////////////////////////

1、閥

2、流量計

3、送水泵

4、壓力變送器

create 'hbase_histtable', 'info'

put 'hbase_histtable' ,'17553973test20210906','info:id','17553973'

put 'hbase_histtable' ,'17553973test20210906','info:AltId','ZH1#YYB_YW_Down'

put 'hbase_histtable' ,'17553973test20210906','info:HistValue','0'

put 'hbase_histtable' ,'17553973test20210906','info:UpdateTime','2021-04-29 09:26:09'

////////////////////////////////////////////////////////////////////////////////////////////////////////

1、下載下傳hive

wget https://downloads.apache.org/hive/hive-1.2.2/apache-hive-1.2.2-bin.tar.gz

tar xvf apache-hive-1.2.2-bin.tar.gz -C /usr/local

2、配置profile

export HIVE_HOME=/usr/local/hive

export PATH=$PATH:$HIVE_HOME/bin

export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.

3、修改配置Hive

配置Hive用于Hadoop環境中，需要編輯hive-env.sh檔案，該檔案放置在 $HIVE_HOME/conf目錄。下面的指令重定向到Hive config檔案夾并複制模闆檔案：

$ cd $HIVE_HOME/conf

$ cp hive-env.sh.template hive-env.sh

通過編輯hive-env.sh檔案添加以下行：

export HADOOP_HOME=/usr/local/hadoop

$ cp hive-default.xml.template hive-site.xml

$ vim hive-site.xml

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://172.20.10.12:3306/metastore</value>

</property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>com.mysql.jdbc.Driver</value>

</property>

<name>javax.jdo.option.ConnectionUserName</name>

</property>

<name>javax.jdo.option.ConnectionPassword</name>

</property>

<name>hive.metastore.schema.verification</name>

<value>false</value>

</property>

cp mysql-connector-java-5.1.36.jar /usr/local/hive/lib

[hadoop@hadoop102 software]$ mv $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.jar $HIVE_HOME/lib/log4j-slf4j-impl-2.10.0.bak

4、啟動Hive

5.1 初始化中繼資料庫

1）登陸MySQL

[hadoop@hadoop102 software]$ mysql -uroot -prhXA@1812

2）建立Hive中繼資料庫

mysql> create database metastore;

mysql> quit;

3）初始化Hive中繼資料庫

[root@SNAPP02 conf]# schematool -initSchema -dbType mysql -verbose

5.2 啟動metastore

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

這是因為在hive-site.xml配置檔案中需要配置system:java.io.tmpdir屬性。

在配置檔案中加入：

<name>system:java.io.tmpdir</name>

</property>

hive指令出現問題Failed with exception Java.io.IOException:java.lang.IllegalArgumentException: java.NET.URI

Failed with exception Java.io.IOException:java.lang.IllegalArgumentException: java.NET.URISyntaxException: Relative path in absolute URI: ${system:user.name}

進入apache-hive-2.1.1-bin檔案夾下的conf檔案下,找到hive-site.xml，具體步驟：

　apache-hive-２.１.1-bin -> conf -> hive-site.xml

　然後找到如下屬性：

<name>hive.exec.local.scratchdir</name>

<value>/home/lch/software/Hive/apache-hive-2.1.1-bin/tmp/${ｓｙstem:user.name}</value>

<description>Local scratch space for Hive jobs</description>

</property>

把它修改成如下：

<name>hive.exec.local.scratchdir</name>

<value>/home/lch/software/Hive/apache-hive-2.1.1-bin/tmp/${user.name}</value>

<description>Local scratch space for Hive jobs</description>

</property>

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

hive 整合hbase

create 'hbase_histtable', 'info'

put 'hbase_histtable' ,'17553973test20210906','info:id','17553973'

put 'hbase_histtable' ,'17553973test20210906','info:AltId','ZH1#YYB_YW_Down'

put 'hbase_histtable' ,'17553973test20210906','info:HistValue','0'

put 'hbase_histtable' ,'17553973test20210906','info:UpdateTime','2021-04-29 09:26:09'

CREATE EXTERNAL TABLE hbase_histtable(key string,id bigint,AltId string,HistValue string,UpdateTime string )

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,info:id,info:AltId,info:HistValue,info:UpdateTime")

TBLPROPERTIES ("hbase.table.name" = "hbase_histtable");

CREATE EXTERNAL TABLE hive_histtable(Rowkey string,id string,AltId string,HistValue string,UpdateTime string )

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,info:id,info:AltId,info:HistValue,info:UpdateTime")

TBLPROPERTIES ("hbase.table.name" = "hbase_histtable");

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

1、下載下傳sqoop

wget http://archive.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

2、解壓

tar xvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /usr/local/

3、配置環境變量

編輯/etc/profile檔案，添加SQOOP_HOME變量，并且将$SQOOP_HOME/bin添加到PATH變量中

export SQOOP_HOME=/usr/local/sqoop

export PATH=${SQOOP_HOME}/bin:$PATH

4、Sqoop配置檔案修改

sqoop-env.sh檔案

cd /usr/local/sqoop/conf

cp sqoop-env-template.sh sqoop-env.sh

編輯這個建立的sqoop-env.sh檔案

export HADOOP_COMMON_HOME=/usr/local/hadoop

export HADOOP_MAPRED_HOME=/usr/local/hadoop

export HBASE_HOME=/usr/local/hbase

5、将MySQL驅動包上載到Sqoop的lib下

cp mysql-connector-java-5.1.36.jar /usr/local/sqoop/lib

6、使用Sqoop檢視MySQL中的資料表

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

說明：基于hadoop營運java.jar檔案時，報The auxService:mapreduce_shuffle does not exist錯誤，具體異常如下：

INFO mapreduce.Job: Task Id : attempt_1461808335315_0001_m_000000_1, Status : FAILED

Container launch failed for container_1461808335315_0001_01_000003 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

解決辦法：

修改yarn-site.xml 檔案，并且要hbase02 和 hbase03 叢集檔案都同步修改，如下：

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

https://blog.csdn.net/qq_38256924/article/details/79850369?utm_medium=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.control&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-2%7Edefault%7ECTRLIST%7Edefault-1.control

sqoop list-tables --username root --password 'rhXA@1812' --connect jdbc:mysql://172.20.10.12:3306/xasnplc?characterEncoding=UTF-8

sqoop eval --connect jdbc:mysql://172.20.10.12:3306/xasnplc --username 'root' --password 'rhXA@1812' --query "SELECT * FROM histtable WHERE ID=15"

sqoop import --connect jdbc:mysql://172.20.10.12:3306/xasnplc --username root --password 'rhXA@1812' --table histtable --hbase-table hbase_histtable --column-family info

sqoop import --connect jdbc:mysql://192.~~.~~.146:3306/yfei --username 'root' --password '你的密碼' --table 'mysql資料庫的表名' --hbase-table 'test' --hbase-row-key '可以是mysql的主鍵' --column-family '剛剛建立的列簇的名：region'

sqoop eval --connect jdbc:mysql://47.~~.~~.146:3306/yf --username 'root' --password '你的密碼' --query "SELECT * FROM USERS WHERE ID='123'"

7、Hbase 資料導入 MySql中的方案為，hive 挂載 hbase表，使用sqoop 導hive 表到 mysql中去

create table bak_histtable(id int,altid string,histvalue string,updatetime string);

sqoop export --connect jdbc:mysql://172.20.10.12:3306/xasnplc --username root --password 'rhXA@1812' --table bak_histtable --input-fields-terminated-by '\001' --export-dir hdfs://hbase01:9000/user/hive/warehouse/bak_histtable

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

mysql 接口表設計

CREATE TABLE `histtable` (

`ID` bigint(20) NOT NULL AUTO_INCREMENT,

`AltId` varchar(30) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,

`HistValue` float NULL DEFAULT NULL,

`UpdateTime` datetime NULL DEFAULT NULL,

PRIMARY KEY (`ID`) USING BTREE

)

PARTITION BY LIST (YEAR(UpdateTime)*10000 + MONTH(UpdateTime)*100 +DAY(UpdateTime))

(

PARTITION p20210831 VALUES IN (20210831),

PARTITION p20210901 VALUES IN (20210901),

PARTITION p20210902 VALUES IN (20210902),

PARTITION p20210903 VALUES IN (20210903),

PARTITION p20210904 VALUES IN (20210904),

PARTITION p20210905 VALUES IN (20210905),

PARTITION p20210906 VALUES IN (20210906)

);

SELECT * FROM histtable PARTITION(p20210831);

SELECT YEAR(DATE_SUB(NOW(),INTERVAL 8 DAY))*10000+MONTH(DATE_SUB(NOW(),INTERVAL 8 DAY))*100+DAY(DATE_SUB(NOW(),INTERVAL 8 DAY))，DATE_ADD(NOW(),INTERVAL 8 DAY);

ALTER TABLE histtable

ADD PARTITION (PARTITION p20210909 VALUES IN (20210909) );

ALTER TABLE histtable DROP PARTITION p20210909 ;

sqoop import --connect jdbc:mysql://172.20.10.12:3306/xasnplc --username root --password 'rhXA@1812' --query "select CONCAT(id,DATE_FORMAT(updatetime,'%Y%m%d%H%i%s')) as keyrow,id,AltId,CONCAT(HistValue,'') as HistValue,CONCAT(UpdateTime,'') as UpdateTime from histtable_test where 1=1 and \$CONDITIONS" --hbase-table hbase_histtable --hbase-row-key keyrow --split-by id --column-family info

INSERT OVERWRITE TABLE hive_histtable SELECT * FROM hive_histtable WHERE updatetime>'2021-09-25';

hive -e "select concat('deleteall \'hbase_histtable\',\'',id,'\'') from hive_histtable where updatetime>'2021-09-25'" > del_temp.txt

hive -e "select concat('deleteall \'hbase_histtable\',\'',rowkey,'\'') from hive_histtable where updatetime>'2021-09-25'" > del_temp.txt

hbase shell del_temp.txt > del.log

hbase shell

truncate 'hbase_histtable'

/////////////////////////////////////////////////////////

insert into msg_to_send(msg_title,msg_content,msg_type,msg_send_to,msg_receiver_name,msg_receiver_id,

event_time,event_place_desc,LGTD,LTTD,in_time,out_time,update_time,msg_state)

SELECT '樞紐 PLC 接口資料異常告警' msg_title,'樞紐histtable 曆史資料接口異常' msg_content,

1 msg_type, 1 msg_send_to,D.user_name msg_receiver_name,D.user_info_num msg_receiver_id,

CURRENT_TIMESTAMP event_time, '樞紐機房' event_place_desc,112.12864100 LGTD,34.72299200 LTTD,CURRENT_TIMESTAMP in_time,

null out_time,CURRENT_TIMESTAMP update_time,0 msg_state

from msg_info_user D

where D.stat =1; /*狀态有效的使用者*/

hbase（分布式nosql資料庫）安裝

繼續閱讀

關系型資料庫與非關系型資料庫1.NoSQL[非關系型資料庫]2.關系型資料庫和非關系型資料庫對比3.NoSQL分類

Linux與Windows安裝Redis

Linux下MongoDB(3.0)的安裝和配置使用者

Base L2: 不止于 Meme Coin，Coinbase的整合優勢或帶來新的影響力

安裝SQL Server2008 r2 提示SQL Server 2005 Express 工具失敗和完全解除安裝SQL Server資料庫操作流程

HBase第二天：HBase的API操作，判斷表存在、建立删除表、擷取表中一行或指定列族資料、向表中插入資料、HBase的wordcount、自定義HBaseMapReduce、Hbase內建Hive第6章 HBase API操作

資料庫時間格式轉換

hadoop實戰多表關聯

MongoDB：資料庫角色權限1. 了解Admin資料庫2. 角色與權限3. 建立角色

hbase thrift C++ 簡單測試

Cloudera Manager HBase Thrift 接口 Go/Python用戶端

Percolator Google的海量資料增量處理系統

redis主從複制與哨兵模式

Redis的快速上手與基本使用

大資料技術原理與應用（最後三天備考了！！！）

ubuntu14.04下安裝hbse1.0.1.1